Thursday, December 10, 2009

create a grd file from DEM

After creating a DEM file using SRTM.py script, now say we need to convert it to a GMT .grd file.

1. convert the binary file to a grd file:
xyz2grd [binary file] -G[output_grd_file] -Zh -I[pixel_size] -R[west]/[south]/[east]/[north]r
- don't forget the r at the end.
- in the -Zh, h is for 2 byte integer. change if needed
2. Sample to a different pixel size and sub-region:
grdsample [input_grd_file] -G[output_grd_file] -I[new_pixel_size] -R[west]/[south]/[east]/[north]r
- if a known number of pixels is desired, say in east-west, use: east = west+[needed_pixel_number - 1]*[new_pixel_size]
3. put zeros instead of null values -9999:
grdmath [input_grd_file] -9999 GT [input_grd_file] MUL = [output_grd_file]
- note that if you sample some of the null pixels will be with different values so use say -500 instead of -9999 as a threshold.

Friday, October 2, 2009

Reading GMT .GRD/NetCDF file using python

When reading GTM file in GRD format you actually read a NetCDF format.
using python and pylab to read it enable matplotlib to plot it.
say we have a grd file named z.grd which contains 3 variables: x,y and z.
reading z values in python is done by the following three lines:

from pylab import *
from scipy.io import netcdf_file as netcdf
data = netcdf('z.grd','r').variables['z'][::-1]

Now data is a numpy array containing z values in the file dimensions.
for more info look at: http://gfesuite.noaa.gov/developer/netCDFPythonInterface.html

Tuesday, September 29, 2009

howto edit multiple files via command line

In order to replace a regexp in multiple files using the command line use:

sed -i -e 's/regexp/new_regexp/' file_name

where file_name can be also a regexp (e.g. *.txt)

Wednesday, July 22, 2009

Hard Drive Resurrection

July 22, 2009: One of my hard drives turned out to be dead.
When I disconnected it from the computer - all is well, the computer is working fine without the missing HD, only I had very important data on it and must resurrect it.
when I connect it - the whole system is freezing So I belive it is something to do with the logic board. I googled it up and came upon this.
Basically, I need another HD with the same properties (manufacturer, model,firmware) and just switch the logic board which should be easy and hopefully work.
I now need to find a similar HD for the replacement.

July 23, 2009: Well, no luck in finding a similar HD and sending the HD to data recovery is more than 150$. Before I spend so much money I would like to check the cheapest option:
Freezing the HD. lets see what happens...

July 23, 2009, supplemental: it worked!
I was able to recover the data after few hours in the freezer. I hardly believed it, but it actually worked. I'll now spend my money buying a larger HD.
wooohooo!

Monday, May 25, 2009

HowTo view a ROI PAC file? (or "replacement for mdx")

The Problem:
I can't get mdx/dgx in order to view my ROI PAC files.
The Solution:
a Python script called roiview (Get RoiView at SourceForge.net. Fast, secure and Free Open Source software downloads) wich is one of my other scripts.
This script uses pylab and can read, display and save images of ROI PAC quite similar to mdx/dgx.
it can handle *.unw, *.cor, *.slc, *.int and *.dem files.
just download, extract and use:
"roiview -h" for manual.
Update! Oct 13, 2010: A new version of RoiView (0.75) is now available at: SourceForge.net. Fast, secure and Free Open Source software downloads

Wednesday, May 13, 2009

HowTo Create a SRTM DEM for ROI PAC Processing

When Processing interferograms, usually, you will also require a DEM. The NASA's SRTM project is freely available from different places in different formats and levels. The latest version for now (IV) is available from CGIAR Consortium for Spatial Information.
in order to ease the DEM production for a research, I wrote a small Python script. The script will create a DEM with 1 deg to each side of the center of scene requested. It will create a ROI PAC header as well. there is an option to create a JPG image of shaded relief of the imag. You will need Python to be installed and also PyLab and PIL modules.

simply download SRTM.py (zip,2.9kb) from my scripts page, change permission to execute, and run it...

Don't forget to credit CGIAR for the data. Citations should be made as follows: Jarvis A., H.I. Reuter, A. Nelson, E. Guevara, 2008, Hole-filled seamless SRTM data V4, International Centre for Tropical Agriculture (CIAT), available from http://srtm.csi.cgiar.org/

Thursday, May 7, 2009

Using HighLearn & Linux

The Problem:
Can't use HighLearn on Linux with firefox.
This e-learning system of some of the academic institutions in Israel is Microsoft explorer based. It is a very unfortunate setting for linux users. I hope they will change the system soon. Probably they won't...

The Solution:
Thanks to a tip from a friend, I google it up and discovered this thread (Hebrew) which suggest using konqueror.
Just install Greasemonkey on your firefox, restart firefox and click "Install" Here.

Note! not all the features are working yet.

Monday, May 4, 2009

Howto view an image in Google Earth

The Problem:
I have an image I wish to view on Google Earth.

The Solution:
I assume you have an image in jpg or tiff or gif format. and you have a world file for it or can create one. Then, all you need to do is to run the image2kml.csh script.
just download (~1kb), apply execution privilege (chmod +x [ScriptFileName])
and run it. or visit my scripts page
Tip:
gif format image can support transparency you can use GIMP or other image manipulation program to turn unwanted parts of the image to be transparent so you can see the Google Earth layover under the image.

Howto share folders from windows to Linux

The Problem:

I use two machines, one is a Linux based and one windows based. I need to share files between them.

The Solution:

One option is to use a network hard drive. It might be quite slow though.

Other option is to use windows share documents option. It is not the safest methods, but surely one of the easiest.

On Windows:

- Right click on the folder to be shared and select “sharing and Security...”








- mark “Share this folder on the network”
- give a network name – best if only one word (it is not case sensitive)
- if you will need to change the files and not only to view them mark also “Allow network users to change my files”








On Linux:

Gui solution:
- go to “Places” -> “Connect
to Server..."
- switch to “Widows Share” on service type
- On “Server:” enter the windows machine IP number
- On “Share:” enter the name of the shared folder as was entered on the Windows sharing options
- if you use a different user name enter your windows user name on “User Name:”
- You can use an alias for this folder on the Linux machine by entering a new name at “Bookmark name:”






Now you can see and use the folder and files.

Or using mount (not Gui):

- create a new mount point (e.g. /mnt/windowsshare)

- Use mount command:
mount -t cifs //serverIP/shareName -o username=WindowsUserName,password=WindowsPassword /mnt/windowsshare

Or edit /etc/fstab file:

mounting windows share:

//XXX.XX.XX.XX/shareName /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]the file should contain the remote system properties:username=[username]password=[userpassword]

Tuesday, April 28, 2009

Whoto know if a long process is done?

The Problem:
I sometimes run a very long process which can even reach up to few days.
Instead of looking and checking if the process has done, I needed a solution to get a message from the process itself.
One way to do it is by using mail like this:
echo "some message" | mail -s "subject of mail" yourmail@wherever
Unfortunately, for some reason it doesn't work for me.
if you have any solution - let me know. I used a workaround...

The Solution (Option I):
Using Thunderbird and cshell and a tip from here (look at the comments), I built a script that can be added to a command line or another script which run's the process.
needed ingredients:
- Thunderbird
- xdotools

usage: simply copy the command below, edit where needed and run:

mail_script.csh [few message body words]

mail_script.csh:
#!/bin/csh
# this will send a mail with subject:"mission accomplished"
# and command line variables (argv) as the message body
# the script require Thunderbird and xdotool installed
# Just change the "to" field in the command below and
# set a program to run it at the end of process
# by Ran Novitsky Nof 2009 @ BGU
set body = `echo $argv`
thunderbird -compose "to='yourmail@wherever',subject='Mission Accomplished',body=$body" & ; sleep 5 ; set WID=`xdotool search --title
"Compose" head -1`; xdotool windowfocus $WID;xdotool key ctrl+Return
 The Solution (Option II):
  Option I needs Thunderbird to be installed and it uses gui so somtimes there are conflicts with user activity. an alternative solution is to install ssmtp (most Linux distros has it as a package). after installing edit /etc/ssmtp/ssmtp.conf file:

most importent lines to edit:
Mailhub - you smtp server (e.g smtp.gmail.com or any other smtp server)
AuthUser - see conf file link above
AuthPass - see conf file link above
AuthMethod - see conf file link above
FromLineOverride - remove the # at the start of line

edit /etc/ssmtp/revaliases:
add a line:
root:[user]@[mailaccount]:[smtpserver]:[port]

and run the script mail_script.csh:

# this will send a mail with subject:"mission accomplished"
# and command line varibles (argv) as the message body
# the script requier ssmtp installed
# Just change the -F , SendTo and "To" fields in the command below and set a program to run it at the end of process
#
# by Ran Novitsky Nof 2010 @ BGU
set body = `echo $argv`
sudo ssmtp -F"name of sender" recipient@mailserver << END
To:recipient@mailserver
Subject:Mission accomplished
$body
END

Tag Clouds

The Problem:
How to replace the tag list (aka labels) to a tag cloud?
The Solution:
Details in here you can see the outcome on the sidebar of my blog.
Tip:
To add a heading to the tags cloud simply add another <div> before the div of widget-content:

new div: <div><h2 class="'title'">Labels Cloud</h2></div> (in red is the title I selected for the gadget, can be anything actually)

original script (only the first few lines):<b:widget id="'Label1'" locked="'false'" title="'Labels'" type="'Label'"><b:includable id="'main'"> [new div in here] <div class="'widget-content'"><div id="'LabelDisplay'/"></div><script language="'javascript'" type="'text/javascript'">

Monday, April 27, 2009

Orbits and other auxiliary data

I use Delft orbital data (ODR) which is available from here. don't forget to use the precise version and to download all .ODR files (make sure it is not .odr after downloading) and the arclist. also, download and install getorb software and link or move the executables to the ROI_PAC binaries directory.

I also use DORIS orbital data for Envisat which is available from here. but will require a user and password. the DORIS data is used when no ODR data is yet to be available for new Envisat images.

for Envisat I also use auxiliary data available from here.

in my lab, all the orbital data is in ~/ROI_PAC_3_0/Delft with directory tree as follows:

in ~/ROI_PAC_3_0/Delft I put the Envisat auxiliary data (INS and XCA files)

  • doris/Envisat/YYYYMM contains orbital data for the year (YYYY) and month (MM)
  • Envisat contains Envisat Delft orbit data (.ODR .res arclist files)
  • ERS1 contains ERS1 Delft orbit data (.ODR .res arclist files)
  • ERS2 contains ERS2 Delft orbit data (.ODR .res arclist files)

the files must be updated from time to time as more images are added to the list.

ROI PAC installation

UPDATE: Oct 9, 2011 (for version 3.0.1):

first download ROI-PAC and test data from open channel Foundation

I will assume the tar files are extracted at home directory (~), but it can be anywhere you wish.

don't forget to grand read/write access to relevant users.
make sure you have at least gcc and gfortran installed on your system.

go to ~/ROI_PAC_3_0_1/ROI_PAC/ directory.

follow steps described in ~/ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC up until line:
which uncompress

make sure you have a TEST DIR testing data to test the compiled program. this directory should be tared but not gzipped using:
gunzip roi_pac_test_dir.tar.gz
this will convert the roi_pac_test_dir.tar.gz to roi_pac_test_dir.tar
then create a directory test-runs in the ROI_PAC directory.
cd test_runs
and run the multitest.sh script:
../contrib/multitest.sh /FULLPATH/roi_pac_testdir.tar /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/share/roi_pac /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILD1/bin  ... ... ... /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILDN/bin

where:
FULLPATH is the path to your home directory (or where ever the install is)
xxxxxx-yyyy is the multibuild date-time stamp
and BUILD1 to BUILDN are all the compilers sub directories in /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/ e.g. defaults, gfortran, gfortran64 etc.

check for the fastest build by checking the batchlog files for each build in test-runs directory.

go to multibuild directory and into installs. get the bin directory from the desired compiler directory and copy it to the ~/ROI_PAC_3_0_1/ROI_PAC directory.

  • note: I moved the bin directory to: ~/ROI_PAC_3_0/ROI_PAC/BIN/LIN
see the ROI_PAC Wiki - Installation for updated installation notes
look also at ROI_PAC Wiki for updates and patches.

now you can use ROI PAC, but keep in mined orbits and other auxiliary data should also be available for processing.


Installing Linux Fedora Core 10


Most of the info can be found here

I will only add some relevant steps to InSAR processing as I use it and steps I found useful.

installing gnome and most of the distro (including tcsh as it is my favorite shell)

- change nautilus to open always in browser
- remove SElinux policy
- remove option in keyboard options - allow mouse to be controlled by keypad
- add vino vnc option to user
- change autologin for user: vi /etc/gdm/gdm.schemas
- add user to sudo:

su --login -c 'visudo'
Now go below the line:
root ALL=(ALL) ALL
press a, and type the following
username ALL=(ALL) ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.

- adding windows fonts:
wget http://www.my-guides.net/en/images/stories/fedora10/msttcore-fonts-2.0-2.noarch.rpm
su -c 'rpm -ivh msttcore-fonts-2.0-2.noarch.rpm'
- open terminal here in nautilus
su -c 'yum install nautilus-open-terminal'
- Enable Num Lock on GNOME startup
su -c 'yum install numlockx'
su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
su -c 'gedit /etc/gdm/Init/Default'
And add the following lines at the end of the file before the exit 0.
if [ -x /usr/bin/numlockx ]; then
/usr/bin/numlockx on
fi

- add to install:
yum extender
yum Fastest Mirror Plugin
GMT
qgis
grace
filezilla
ipython
KDocker
google-earth
wine
g77
bluecurve icons
gnash - for flash viewing

- setting /etc/X11/xorg.conf for display with no monitor attached:
make sure to use a video card which is not onboard (or perhaps this is only valid for the 965 intel mobo?)

- edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]

- adding python modules
for gui: http://easygui.sourceforge.net/index.html
scipy
numpy
matplotlib
ipython
pylab

in a case of emergency: enabling write on a system error:
mount -t ext3 /dev/sda1 / -o remount,rw

- installing ROI PAC
- adding axillary data

Working on two machines at once

The Problem:
On my work I use two computers, one is Linux based which carry out all the heavy processing and another is windows based for viewing results, office related jobs and other Microsoft compatibility issues.

My problem was I had two mice, two keyboard and two screens. it is very confusing and tiring to switch from one to another.

The Solution:
The Linux distro comes with a wonderful component - VINO SERVER a VNC server which connects to your display 0 (the main display).
so I installed REALVNC viewer on my windows computer, connected one screen to the DVI exit and one to the D-sub (VGA) exit on the graphic card.
now, all need to do is to enter the Linux machine via VNC, drag the window to the second screen and use "Full Screen" view. This allows me to use one set of keyboard and mouse and move from one machine to another just by moving from one screen to another.

Tips:
  • autologin:
    Since you can't "see" the Linux machine without connecting to it, you can tweak the bootloader to load you user automatically:
    edit /etc/gdm/custom.conf (as superuser if needed) and change:
    [daemon]
    TimedLoginEnable=true
    TimedLogin=yourusername
    TimedLoginDelay=0
  • ssh first:
    if you don't want to use autologin for security or other reason, you can login via ssh and start a gnome session and then use the VNC viewer. just enable ssh daemon and vino server.
  • othere VNC servers:
    you can always use different VNC server and connect to other displays then 0.
  • graphic card:
    I tried to use this method with an onboard G. card and only got low resolution screen size.
    it might be my mother board (intel DQ965GF) or bad X settings but using a GeForce card I managed to use my desired resolution with no problems.

Welcom to My Blog.

My name is Ran Novitsky Nof. I currently work at the Geophysical Institute of Israel.

This blog's first aim is to serve as a holding place for my ideas, thoughts, tips and workflows during my research and sharing it with the world - if only one person would benefit from something here - I did my share.

Additional Info may be found at my personal homepages : ran.rnof.info