Dead Sea rising: how the lowest place on Earth is getting (slightly) higher - GeoSpace - AGU Blogosphere

See a blog post about my recent paper:
Dead Sea rising: how the lowest place on Earth is getting (slightly) higher - GeoSpace - AGU Blogosphere

Howto run a process automatically when a certain mail is downloaded by Thunderbird

In order to run a file whenever a certain mail arrives to my inbox I created a filter using the Thunderbird add-on FiltaQuilla. The add-on expand the messages filtering options and add an option to "run a file". when a certain mail match the filter, Thunderbird will run the file specified. I use a python code - code.py so in the filter configuration under Linux (Fedora core 14) I use this line:

/usr/bin/python,/home/UserName/code.py,-a

The line is a comma separated 3 parts line. The first part is the python interperter, the second is the script name to run and the last part is an argument needed for the script to run. Omitting the first part and the filter wont work.

HowTo Mosaic ASTER Global Digital Elevation Model (GDEM)

A second version of ASTER GDEM was released in Oct. 17 2011 by The Ministry of Economy, Trade and Industry of Japan (METI) and the National Aeronautics and Space Administration (NASA). The GDEM is in a 30m pixel resolution and 1x1 degree tiles. The data is distributed as zipped GeoTIFF files. When a larger than 1x1 deg. DEM is needed for Interferometry processing, you can use my GDEM.py script available from my script page. This python script will need a system with pylab and PIL python modules.

Simply:

Download the GDEM zipped files (see here on how to do that).

Unzip the zip files in the working directory.

The working directory should contain sub-directories containing the GeoTIFF files.

Run the script giving -h for help or the final DEM name as arguments.

The Final DEM is aimed for using as DEM in ROI-PAC interferometry processing.

If a -g is added as an argument, the produced DEM will be suitable for interferometric processing using Gamma software.

Create a photo collage using a single command-line

In order to create a contact-sheet or a collage of many files under linux, I prefer to use a single command-line. I create a directory with all the images and use ImageMagick montage function:

montage -font Arial-Narrow -pointsize 10 -label '%t' -resize 100x150 *.jpg -geometry +3+3 -tile 4x4 -shadow collage.jpg

HowTo use a computer without a screen attached

The Problem:
I have a Linux machine (Fedora 14)  with no screen attached. I view the graphics via VNC server. I had to adjust the system to use the Nvidia driver and tweak it for no screen.

The solution:
First we need to install the Nvidia driver. Then, we need to adjust the X configuration for no screen and our preferred resolution.

for more details see: http://forums.fedoraforum.org/showthread.php?t=204752

1. Update the kernel:

su
yum update kernel*
reboot

2. Add rpmfusion to your repository.

su
rpm -Uvh  http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm   http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

3. Depending on your card add nvidia akmod to the kernel and the proper driver. for a Gforce 9500GT on my 64bit machine:

yum install akmod-nvidia-260.19.36-1.fc14.x86_64 xorg-x11-drv-nvidia-libs-260.19.36-1.fc14.x86_64

4. Edit the xorg.conf file. don't forget to  adjust the Nvidia card for no screen attached by adding:
 Option "ConnectedMonitor" "CRT"
to the device section and adjust the screen resolution under the Screen section:

Section "Device"
    Identifier     "Card0"
    Driver         "nvidia"
	Option "ConnectedMonitor" "CRT"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Card0"
    Monitor        "Monitor0"
    SubSection     "Display"
        Viewport    0 0
        Modes      "1280x1024"
    EndSubSection
EndSection

Daichi - ALOS : finding Images and Baseline Calculation

Using JAXA satellite "DAICHI" (ALOS) for SAR Interferometry might raise the need for calculating perpendicular baselines. One can use AUIG site to locate the needed images for interferometry and use PALSAR baseline calculator CalcBP for calculating perpendicular baselines. more info is available here.

Installing Linux Fedora Core 14

Well, after a long time using Fedora 11 I'm moving on to Fedora 14.
Here I'll detail the steps I used for the installation.
I generally follow my Fedora 10 installation guide with some changes of course.

LiveCD:
I use a liveCD (download page) to install.
after loading the system via cd,
I use the "Install to Hard Drive" icon.
next,next,next.
Giving the Hostname, setting location, giving a good root password.

File System:
Since I use several hard drives I need to configure the file system. I use one 1.5TB disk as my home, another one for data and processing and one for the system so all the installation will only deal with the last one. later I'll mount all the rest of the disks.
so the type of installation for me is "Replace Existing Linux System(s)", but with "Review and modify layout" switch on and I select the system disk as "Install Target Devices".
Next I change the partitions and mount points of all disks as needed.
Finally, The boot loader is set up and the system is installed on the hard drive.

Updates and softwares:
As a first step few softwares should be installed and update (this might take forever):

system-config-network
yumex
gconf-editor
gedit-plugins
gthumb
nautilus0open-terminal
tango-icon-theme
tango-icon-extras
fonts for Hebrew
all stix fonts
cm-lgc fonts
blender
gimp
gv
ImageMagick
inkscape
filezilla
thunderbird
emacs
open office
vim-enhanced
gcc
gmt
grace
HDF
HDF5
lapack
netcdf
numpy
python support for open office
compat-gcc
glade3
ipython
pango-devel
wget
tcsh
system-config-samba
system-config-users
system-config-nfs
dos2unix
selinux configuration gui
matplotlib
scipy
scitools
kdocker
numlockx
bluecurve icons
ssmtp

Customizing:
remove SElinux policy
disable filrewall
configure remote desktop
add user to sudo:

su --login -c 'visudo'
Now go below the line:
root ALL=(ALL) ALL
press a, and type the following
username ALL=(ALL) ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.

change autologin for user:
sudo vi /etc/gdm/custom.conf
under [daemon], add:
AutomaticLoginEnable=true
AutomaticLogin=yourUserName

edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]

Enable Num Lock on GNOME startup:
su -c 'yum install numlockx'
su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
su -c 'gedit /etc/gdm/Init/Default'
And add the following lines at the end of the file before the exit 0.
if [ -x /usr/bin/numlockx ]; then
/usr/bin/numlockx on
fi

set the samba server

set the mail server:
sudo vi /etc/ssmtp/ssmtp.conf
change the line:
mailhub=mail
to:
mailhub=[your outgoing mail server]

Set the screen resolution for vnc and no monitor:
create xorg.conf file
sudo Xorg :1 -configure
sudo cp /root/xorg.conf.new /etc/X11/xorg.conf
and add lines to the xorg.conf file:

Section "Monitor"
Identifier   "Monitor0"
ModelName    "LCD Panel 1280x1024"
HorizSync    31.5 - 64.0
VertRefresh  56.0 - 65.0
Option    "dpms"
EndSection

Section "Device"
        ### Available Driver options are:-
        ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
        ### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
        ### <percent>: "<f>%"
        ### [arg]: arg optional
        #Option     "SWcursor"           # [<bool>]
        #Option     "HWcursor"           # [<bool>]
        #Option     "NoAccel"             # [<bool>]
        #Option     "ShadowFB"           # [<bool>]
        #Option     "UseFBDev"           # [<bool>]
        #Option     "Rotate"             # [<str>]
        #Option     "VideoKey"           # <i>
        #Option     "FlatPanel"           # [<bool>]
        #Option     "FPDither"           # [<bool>]
        #Option     "CrtcNumber"         # <i>
        #Option     "FPScale"             # [<bool>]
        #Option     "FPTweak"             # <i>
        #Option     "DualHead"           # [<bool>]
Identifier  "Card0"
Driver      "vesa"
BusID       "PCI:1:0:0"
EndSection

Section "Screen"
Identifier "Screen0"
Device     "Card0"
Monitor    "Monitor0"
SubSection "Display"
Viewport   0 0
Depth     24
Modes "1280x1024"
EndSubSection
EndSection

set the panels and desktop background as you like it.

Install RPMfusion repositories following: www.rpmfusion.org/Configuration

add true type fonts from windows

RoiView - Explore InSAR data and more

RoiView was originaly designed to replace DGX/MDX software which require bureaucracy to obtain. This software was used in order to view ROI-PAC interferometry processing results but producing an image from file was not so strait forward. In time support was added for ERDAS ER Mapper header files and for user provided parameters with or without header files.

Current RoiView version (0.75) is available for download at
See also RoiView's help files

Screenshot:

HowTo Parse HTML text using python.

I needed to to get the text of a specific div elemnt in an html file. I tried to use python's standard library modules for markup processing such as htmllib etc. but I couldn't figure how to use them. I've created my own module just for getting the text from an html element:

#!/bin/env python
from htmlentitydefs import entitydefs as ent
import string
# This module enable you to extract text from a certian HTML element
#
# by Ran Novitsky Nof, 2010
# ran.nof@gmail.com
#
# example of use:
# say we want to get the text in an element of type tag (e.g. 'div','a','span' etc.)
# who has an attribute key (e.g. "id","class","href" etc.) with a value of val
# for example in order to extract the text of a div element with id of textdiv from a file htmlfile.html:
#
# <html>
# <head>
#   :
# </head>
# <body>
#   :
# <div "id"="textdiv">I will not buy this <a href="spam">record</a> it is scratched. </div>
#   :
# </body>
# </html>
#
# use:
# from htmlparser import Parser
# htmlfile='htmlfile.html'
# tag,key,val = ('div','id','textdiv')
# text=Parser(htmlfile).getText(tag,key,val)
# print(text)
#
class Element():
  def __init__(self):
    self.startTag = -1
    self.endTag = -1
    self.attrib = {}
    self.keys = self.attrib.keys()
    self.innerHTML = ''
    self.tag = ''
    self.start = -1
    self.end = -1

class Parser():
  def __init__(self,infile):
    self._root = open(infile).read()
    self._root = self._root[self._root.find('<body'):self._root.find('</body')]
    self._root = self._root[self._root.find('>')+1:]
    self.tags = set()
    i,j=0,0
    self.tagstarts = {}
    self.tagends = {}
    self.elements = []
    while i<len(self._root): 
      i = self._root.find('<',i)
      j = self._root.find('>',i)
      if not j>i: break
      tag = self._root[i+1:j].split()[0]     
      if tag.startswith('/'):
        tag = tag[1:]
        self.tagends[tag][-1-self.tagends[tag][::-1].index(None)]=((i,j))
      else:
        self.tags.add(tag)
        if tag in self.tagstarts:
          self.tagstarts[tag].append((i,j))
          self.tagends[tag].append(None)
        else:
          self.tagstarts[tag]=[(i,j)]
          self.tagends[tag]=[None]
      i=j+1
    self.getElements()
  def getElements(self):
    for tag in self.tags:
      if not tag in ['img']:
        for i in range(len(self.tagstarts[tag])):
          element = Element()
          element.startTag = self.tagstarts[tag][i][0]
          element.endTag = self.tagstarts[tag][i][1]
          tagData = self._root[element.startTag+1:element.endTag].replace("\"","")
          element.tag = tag
          element.attrib=dict([a.split('=') for a in tagData.split() if '=' in a])
          element.start = element.startTag
          element.end = self.tagends[tag][i][1]
          element.innerHTML = (self.tagstarts[tag][i][1]+1,self.tagends[tag][i][0])
          self.elements.append(element)
  def getText(self,tag,key,val):
    element = [element for element in self.elements if element.tag==tag and element.attrib[key]==val]
    if len(element):
      element = element[0]
      start,end = element.innerHTML
      text = self._root[start:end]
      i,j=0,0
      while i<len(text):
        i  = text.find('<',i)
        j = text.find('>',i)
        if i<0 or j<0: break
        text = text[:i]+text[j+1:]
      i,j=-1,-1   
      while i<len(text):
        i  = text.find('&',i+1)
        j = text.find(';',i+1)
        if i<0 or j<0: break
        if text[i+1:j] in ent.keys(): text = text[:i]+ent[text[i+1:j]]+text[j+1:]
    else:
      text=None 
    return text

Note the code does not check if the html code is correct. it also work only for the body part, and ignore img tags.
   

SSH with no password

When you wish to login to a remote system through an ssh session without using any password you'll need to produce a key on the local computer and on the remote computer in order to allow password - free entrance.
first you'll need to produce the key in the local computer:
ssh-keygen -t dsa -b 1024 -P ""
press Enter on the questions followed by the command.
the output will be something like:
Your identification has been saved in /home/yourname/.ssh/id_dsa.
Your public key has been saved in /home/yourname/.ssh/id_dsa.pub.
The key fingerprint is:
4f:9e:4c:17:8f:19:37:32:bc:45:4f:2a:a2:8e:1c:15 yourname@your.loacl.machine
The key's randomart image is:
+--[ DSA 1024]----+
|           .o    |
|           .oo.  |
|          . o+   |
|      o    *.+. .|
|       .E . +  . |
|      o  .+.. . .|
|       +o.o. +   |
|       o+E       |
|       o=.       |
+-----------------+

This will produce two files in you .ssh directory: id_dsa and id_dsa.pub

cat id_dsa.pub into ~/.ssh/authorized_keys on the remote system you want to log into.
change mod to 600:
chmod 600 ~/.ssh/authorized_keys

(source of the tip)

Howto export points to a Google Earth KML file

The problem:
A table of xyz data points can be viewed spatially with different methods:
GMT and ArcView are just a couple of options. I often get tables of spatial data points in Microsoft Excel (xp ver.) format or CSV of some kind. An easy (and free) way to visualize the data points on Windows would be to export the data and view it on Google Earth.
The solution:
points2kml (115kb).
In order to ease the work-flow I wrote a VBA macro add-in for Excel that exports selected fields (longitude, latitude, name and data) to a Google Earth KML file. the point2kml add-in and can be found on my scripts page.

How to format a hard drive on Linux

The Problem:
How to format a hard drive on Linux.

The Solution:
in order to format a HD on Linux you'll need super user privileges.
you can use sudo if you have sudo privilages or in the terminal
switch to super user:

>su
(give super user pwd)
and ignore the sudo in the following commands. if you're system can't find the commands while in su mode, try to add /sbin/ before the command.
follow the steps below: marked with •
run commnads marked with >
commands output looks like this.
Warning: formatting your HD will erase all data on it!

• Connect the HD to the computer, make sure it's on. 
• get the hard drive's (device) name on your system:

>df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/vg_insarlablin-lv_root

                      908G   77G  786G   9% /

/dev/sda1             194M   31M  153M  17% /boot

/dev/sdb1             1.4T  1.1T  193G  86% /home/novitsky

/dev/sdc1             1.4T  1.1T  193G  86% /home/novitsky/data

Now lets say our HD is sdb1.

• lets build it's partition:

>sudo fdisk /dev/sdb

The number of cylinders for this disk is set to 182401.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): 

• pressing m will get this menu:

Command action

   a   toggle a bootable flag

   b   edit bsd disklabel

   c   toggle the dos compatibility flag

   d   delete a partition

   l   list known partition types

   m   print this menu

   n   add a new partition

   o   create a new empty DOS partition table

   p   print the partition table

   q   quit without saving changes

   s   create a new empty Sun disklabel

   t   change a partition's system id

   u   change display/entry units

   v   verify the partition table

   w   write table to disk and exit

   x   extra functionality (experts only)

• pressing p will print the current partition:

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes

255 heads, 63 sectors/track, 182401 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Disk identifier: 0x000923e6

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1               1      182401  1465136001   83  Linux

• pressing d will delete the current partition. depending on the partition you want to delete (this case 1) press the number of the partition to be deleted. if all the disk should be formatted and there are several partitions repeat pressing d and the number of the partitions until you delete them all.
• pressing n will create a new partition.
• pressing p will create a primary partition. select 1 as the number of the partition. use the defaults (just press Enter) for the start and end cylinders.
• pressing t will set the file system type.
• pressing l will list the available file systems.
• select 83 if you want a Linux file system
• pressing w will write and exit to shell.

Now we will create a new file system on the new partition:
> sudo mkfs -t ext2 /dev/hdb1
remember to change the number at the end if your HD device number is different.
if your using different fs than ext2 (newer Linux systems have ext3 and fc11 has ext4) replace the ext2 in the command with the correct fc name.
the mkfs may take some time to run (depending on the computer and HD size).

To add a label to the partition use:
e2label /dev/sdb1 NEWLABEL
where sdb should be changed to the correct device and NEWLABEL changed to the new label.

Now add the device to the fstab file in order to mount it:
> sudo vi /etc/fstab
and add a line like:
LABEL=NEWLABEL MOUNTPOINT ext3 defaults 1 2
where NEWLABEL is the disk labe and MOUNTPOINT is where you want to mount the disk.
mount it using:
> sudo mount -a
Note you might need to change permissions and owner of the mount point:
> sudo chown username:usergroop MOUNTPOINT
when finnished - reboot.

Moving memory from swap (page) to RAM

When using more memory than your RAM Linux will use swap which is placed on your hard drive and is much slower. Sometimes after using swap there might be a situation where the RAM is free but some of your data is on the swap. There is a way to relocate the swap data to the RAM.
more details and a nice script to do is are available at: Ubuntu Community Documentation Swap FAQ page.

sudo swapoff -a 

sudo swapon -a

The commands simply shut off swap and turning it on back again.
You'll have to make sure there is enough space on the RAM to accommodate the Swap data, this can be done using:

free 

the output will show you the used space of Swap and free space of RAM:

             total       used       free     shared    buffers     cached
Mem:       8047896     772268    7275628          0      19540     129372
-/+ buffers/cache:     623356    7424540
Swap:     10108920          0   10108920

The example above show that no data (0) is stored on Swap.

Using hillshade image as intensity (improved matplotlib shade)

Matplotlib module enables hillshade method (shade) using a LightSource class (v 0.99).
The problem is it uses the data itself as intensity and data. It is very useful for viewing a DEM but sometimes you would like the DEM as intensity underlying some other data. Another problem is that the shade method is producing a very light colored image sometimes even white where intensity is high.
I used as an example a DEM derived from SRTM v4 data acquired at the International  Centre for Tropical  Agriculture (CIAT - http://srtm.csi.cgiar.org) the hillshade production was made using LightSource class with azimuth of 165 deg. and altitude of 45 deg.)

DEM - gist_earth color scheme

Hill-shade (azdeg-165,altdeg-45)

matplotlib shade method

My shade method

The difference in the shading colors derived from the method used to produce it. While the matplotlib method uses "hard light" method I use a "soft light" method. the matplotlib is converting the RGB colors to HSV and then calculate the new saturation and value according to the intensity. I use a formula based on the description of ImageMagick's pegtop_light.which is much faster as it is a single formula. Another advantage is the option to use a separate layer as the intensity and another as the data used for colors.
The modified functions are hillshade and set_shade as follows:

#!/bin/env python
from pylab import *
def set_shade(a,intensity=None,cmap=cm.jet,scale=10.0,azdeg=165.0,altdeg=45.0):
''' sets shading for data array based on intensity layer
  or the data's value itself.
inputs:
  a - a 2-d array or masked array
  intensity - a 2-d array of same size as a (no chack on that)
                    representing the intensity layer. if none is given
                    the data itself is used after getting the hillshade values
                    see hillshade for more details.
  cmap - a colormap (e.g matplotlib.colors.LinearSegmentedColormap
              instance)
  scale,azdeg,altdeg - parameters for hilshade function see there for
              more details
output:
  rgb - an rgb set of the Pegtop soft light composition of the data and 
           intensity can be used as input for imshow()
based on ImageMagick's Pegtop_light:
http://www.imagemagick.org/Usage/compose/#pegtoplight'''
  if intensity is None:
# hilshading the data
    intensity = hillshade(a,scale=10.0,azdeg=165.0,altdeg=45.0)
  else:
# or normalize the intensity
    intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
# get rgb of normalized data based on cmap
  rgb = cmap((a-a.min())/float(a.max()-a.min()))[:,:,:3]
# form an rgb eqvivalent of intensity
  d = intensity.repeat(3).reshape(rgb.shape)
# simulate illumination based on pegtop algorithm.
  rgb = 2*d*rgb+(rgb**2)*(1-2*d)
  return rgb

def hillshade(data,scale=10.0,azdeg=165.0,altdeg=45.0):
  ''' convert data to hillshade based on matplotlib.colors.LightSource class.
    input:
         data - a 2-d array of data
         scale - scaling value of the data. higher number = lower gradient
         azdeg - where the light comes from: 0 south ; 90 east ; 180 north ;
                      270 west
         altdeg - where the light comes from: 0 horison ; 90 zenith
    output: a 2-d array of normalized hilshade
'''
  # convert alt, az to radians
  az = azdeg*pi/180.0
  alt = altdeg*pi/180.0
  # gradient in x and y directions
  dx, dy = gradient(data/float(scale))
  slope = 0.5*pi - arctan(hypot(dx, dy))
  aspect = arctan2(dx, dy)
  intensity = sin(alt)*sin(slope) + cos(alt)*cos(slope)*cos(-az - aspect - 0.5*pi)
  intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
  return intensity

Example of use:
One can save the code to a file named say: shading.py
now say we have a 4 byte float DEM data in a 560 lines 420 samples binary file. in a python code:

from pylab import *
from shading import set_shade
from shading import hillshade
dem = fromfile('DEM.dem',dtype=float32).reshape(560,420)
rgb = set_shade(dem,cmap=cm.gist_earth)
imshow(rgb)

will produce the "my shade method" image as above.

say we have a data to be plot using the DEM data as intensity:

replace the line before last with:
rgb = set_shade(data,intensity=hillshade(dem),cmap=cm.gist_earth)

create a grd file from DEM

After creating a DEM file using SRTM.py script, now say we need to convert it to a GMT .grd file.

1. convert the binary file to a grd file:
xyz2grd [binary file] -G[output_grd_file] -Zh -I[pixel_size] -R[west]/[south]/[east]/[north]r
- don't forget the r at the end.
- in the -Zh, h is for 2 byte integer. change if needed
2. Sample to a different pixel size and sub-region:
grdsample [input_grd_file] -G[output_grd_file] -I[new_pixel_size] -R[west]/[south]/[east]/[north]r
- if a known number of pixels is desired, say in east-west, use: east = west+[needed_pixel_number - 1]*[new_pixel_size]
3. put zeros instead of null values -9999:
grdmath [input_grd_file] -9999 GT [input_grd_file] MUL = [output_grd_file]
- note that if you sample some of the null pixels will be with different values so use say -500 instead of -9999 as a threshold.

Reading GMT .GRD/NetCDF file using python

When reading GTM file in GRD format you actually read a NetCDF format.
using python and pylab to read it enable matplotlib to plot it.
say we have a grd file named z.grd which contains 3 variables: x,y and z.
reading z values in python is done by the following three lines:

from pylab import *
from scipy.io import netcdf_file as netcdf
data = netcdf('z.grd','r').variables['z'][::-1]

Now data is a numpy array containing z values in the file dimensions.
for more info look at: http://gfesuite.noaa.gov/developer/netCDFPythonInterface.html

howto edit multiple files via command line

In order to replace a regexp in multiple files using the command line use:

sed -i -e 's/regexp/new_regexp/' file_name

where file_name can be also a regexp (e.g. *.txt)

Hard Drive Resurrection

July 22, 2009: One of my hard drives turned out to be dead.
When I disconnected it from the computer - all is well, the computer is working fine without the missing HD, only I had very important data on it and must resurrect it.
when I connect it - the whole system is freezing So I belive it is something to do with the logic board. I googled it up and came upon this.
Basically, I need another HD with the same properties (manufacturer, model,firmware) and just switch the logic board which should be easy and hopefully work.
I now need to find a similar HD for the replacement.

July 23, 2009: Well, no luck in finding a similar HD and sending the HD to data recovery is more than 150$. Before I spend so much money I would like to check the cheapest option:
Freezing the HD. lets see what happens...

July 23, 2009, supplemental: it worked!
I was able to recover the data after few hours in the freezer. I hardly believed it, but it actually worked. I'll now spend my money buying a larger HD.
wooohooo!

HowTo view a ROI PAC file? (or "replacement for mdx")

The Problem:
I can't get mdx/dgx in order to view my ROI PAC files.
The Solution:
a Python script called roiview (Get RoiView at ) wich is one of my other scripts.
This script uses pylab and can read, display and save images of ROI PAC quite similar to mdx/dgx.
it can handle *.unw, *.cor, *.slc, *.int and *.dem files.
just download, extract and use:
"roiview -h" for manual.
Update! Oct 13, 2010: A new version of RoiView (0.75) is now available at:

HowTo Create a SRTM DEM for ROI PAC Processing

When Processing interferograms, usually, you will also require a DEM. The NASA's SRTM project is freely available from different places in different formats and levels. The latest version for now (IV) is available from CGIAR Consortium for Spatial Information.
in order to ease the DEM production for a research, I wrote a small Python script. The script will create a DEM with 1 deg to each side of the center of scene requested. It will create a ROI PAC header as well. there is an option to create a JPG image of shaded relief of the imag. You will need Python to be installed and also PyLab and PIL modules.

simply download SRTM.py (zip,2.9kb) from my scripts page, change permission to execute, and run it...

Don't forget to credit CGIAR for the data. Citations should be made as follows: Jarvis A., H.I. Reuter, A. Nelson, E. Guevara, 2008, Hole-filled seamless SRTM data V4, International Centre for Tropical Agriculture (CIAT), available from http://srtm.csi.cgiar.org/

Using HighLearn & Linux

The Problem:
Can't use HighLearn on Linux with firefox.
This e-learning system of some of the academic institutions in Israel is Microsoft explorer based. It is a very unfortunate setting for linux users. I hope they will change the system soon. Probably they won't...

The Solution:
Thanks to a tip from a friend, I google it up and discovered this thread (Hebrew) which suggest using konqueror.
Just install Greasemonkey on your firefox, restart firefox and click "Install" Here.

Note! not all the features are working yet.

Howto view an image in Google Earth

The Problem:
I have an image I wish to view on Google Earth.

The Solution:
I assume you have an image in jpg or tiff or gif format. and you have a world file for it or can create one. Then, all you need to do is to run the image2kml.csh script.
just download (~1kb), apply execution privilege (chmod +x [ScriptFileName])
and run it. or visit my scripts page
Tip:
gif format image can support transparency you can use GIMP or other image manipulation program to turn unwanted parts of the image to be transparent so you can see the Google Earth layover under the image.

Howto share folders from windows to Linux

The Problem:

I use two machines, one is a Linux based and one windows based. I need to share files between them.

The Solution:

One option is to use a network hard drive. It might be quite slow though.

Other option is to use windows share documents option. It is not the safest methods, but surely one of the easiest.

On Windows:

- Right click on the folder to be shared and select “sharing and Security...”

- mark “Share this folder on the network”
- give a network name – best if only one word (it is not case sensitive)
- if you will need to change the files and not only to view them mark also “Allow network users to change my files”

On Linux:

Gui solution:
- go to “Places” -> “Connect
to Server..."
- switch to “Widows Share” on service type
- On “Server:” enter the windows machine IP number
- On “Share:” enter the name of the shared folder as was entered on the Windows sharing options
- if you use a different user name enter your windows user name on “User Name:”
- You can use an alias for this folder on the Linux machine by entering a new name at “Bookmark name:”

Now you can see and use the folder and files.

Or using mount (not Gui):

- create a new mount point (e.g. /mnt/windowsshare)

- Use mount command:
mount -t cifs //serverIP/shareName -o username=WindowsUserName,password=WindowsPassword /mnt/windowsshare

Or edit /etc/fstab file:

mounting windows share:

//XXX.XX.XX.XX/shareName /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]the file should contain the remote system properties:username=[username]password=[userpassword]

Whoto know if a long process is done?

The Problem:
I sometimes run a very long process which can even reach up to few days.
Instead of looking and checking if the process has done, I needed a solution to get a message from the process itself.
One way to do it is by using mail like this:
echo "some message" | mail -s "subject of mail" yourmail@wherever
Unfortunately, for some reason it doesn't work for me.
if you have any solution - let me know. I used a workaround...

The Solution (Option I):
Using Thunderbird and cshell and a tip from here (look at the comments), I built a script that can be added to a command line or another script which run's the process.
needed ingredients:
- Thunderbird
- xdotools

usage: simply copy the command below, edit where needed and run:

mail_script.csh [few message body words]

mail_script.csh:

#!/bin/csh
# this will send a mail with subject:"mission accomplished"
# and command line variables (argv) as the message body
# the script require Thunderbird and xdotool installed
# Just change the "to" field in the command below and
# set a program to run it at the end of process
# by Ran Novitsky Nof 2009 @ BGU
set body = `echo $argv`
thunderbird -compose "to='yourmail@wherever',subject='Mission Accomplished',body=$body" & ; sleep 5 ; set WID=`xdotool search --title
"Compose" head -1`; xdotool windowfocus $WID;xdotool key ctrl+Return

 The Solution (Option II):
  Option I needs Thunderbird to be installed and it uses gui so somtimes there are conflicts with user activity. an alternative solution is to install ssmtp (most Linux distros has it as a package). after installing edit /etc/ssmtp/ssmtp.conf file:

most importent lines to edit:
Mailhub - you smtp server (e.g smtp.gmail.com or any other smtp server)
AuthUser - see conf file link above
AuthPass - see conf file link above
AuthMethod - see conf file link above
FromLineOverride - remove the # at the start of line

edit /etc/ssmtp/revaliases:
add a line:
root:[user]@[mailaccount]:[smtpserver]:[port]

and run the script mail_script.csh:

# this will send a mail with subject:"mission accomplished"
# and command line varibles (argv) as the message body
# the script requier ssmtp installed
# Just change the -F , SendTo and "To" fields in the command below and set a program to run it at the end of process
#
# by Ran Novitsky Nof 2010 @ BGU
set body = `echo $argv`
sudo ssmtp -F"name of sender" recipient@mailserver << END
To:recipient@mailserver
Subject:Mission accomplished
$body
END

Tag Clouds

The Problem:
How to replace the tag list (aka labels) to a tag cloud?
The Solution:
Details in here you can see the outcome on the sidebar of my blog.
Tip:
To add a heading to the tags cloud simply add another <div> before the div of widget-content:

new div: <div><h2 class="'title'">Labels Cloud</h2></div> (in red is the title I selected for the gadget, can be anything actually)

original script (only the first few lines):<b:widget id="'Label1'" locked="'false'" title="'Labels'" type="'Label'"><b:includable id="'main'"> [new div in here] <div class="'widget-content'"><div id="'LabelDisplay'/"></div><script language="'javascript'" type="'text/javascript'">

Orbits and other auxiliary data

I use Delft orbital data (ODR) which is available from here. don't forget to use the precise version and to download all .ODR files (make sure it is not .odr after downloading) and the arclist. also, download and install getorb software and link or move the executables to the ROI_PAC binaries directory.

I also use DORIS orbital data for Envisat which is available from here. but will require a user and password. the DORIS data is used when no ODR data is yet to be available for new Envisat images.

for Envisat I also use auxiliary data available from here.

in my lab, all the orbital data is in ~/ROI_PAC_3_0/Delft with directory tree as follows:

in ~/ROI_PAC_3_0/Delft I put the Envisat auxiliary data (INS and XCA files)

• doris/Envisat/YYYYMM contains orbital data for the year (YYYY) and month (MM)
• Envisat contains Envisat Delft orbit data (.ODR .res arclist files)
• ERS1 contains ERS1 Delft orbit data (.ODR .res arclist files)
• ERS2 contains ERS2 Delft orbit data (.ODR .res arclist files)

the files must be updated from time to time as more images are added to the list.

ROI PAC installation

UPDATE: Oct 9, 2011 (for version 3.0.1):

first download ROI-PAC and test data from open channel Foundation

I will assume the tar files are extracted at home directory (~), but it can be anywhere you wish.

don't forget to grand read/write access to relevant users.
make sure you have at least gcc and gfortran installed on your system.

go to ~/ROI_PAC_3_0_1/ROI_PAC/ directory.

follow steps described in ~/ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC up until line:
which uncompress

make sure you have a TEST DIR testing data to test the compiled program. this directory should be tared but not gzipped using:
gunzip roi_pac_test_dir.tar.gz
this will convert the roi_pac_test_dir.tar.gz to roi_pac_test_dir.tar
then create a directory test-runs in the ROI_PAC directory.
cd test_runs
and run the multitest.sh script:
../contrib/multitest.sh /FULLPATH/roi_pac_testdir.tar /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/share/roi_pac /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILD1/bin  ... ... ... /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILDN/bin

where:
FULLPATH is the path to your home directory (or where ever the install is)
xxxxxx-yyyy is the multibuild date-time stamp
and BUILD1 to BUILDN are all the compilers sub directories in /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/ e.g. defaults, gfortran, gfortran64 etc.

check for the fastest build by checking the batchlog files for each build in test-runs directory.

go to multibuild directory and into installs. get the bin directory from the desired compiler directory and copy it to the ~/ROI_PAC_3_0_1/ROI_PAC directory.

• note: I moved the bin directory to: ~/ROI_PAC_3_0/ROI_PAC/BIN/LIN

see the ROI_PAC Wiki - Installation for updated installation notes
look also at ROI_PAC Wiki for updates and patches.

now you can use ROI PAC, but keep in mined orbits and other auxiliary data should also be available for processing.

Installing Linux Fedora Core 10

Most of the info can be found here

I will only add some relevant steps to InSAR processing as I use it and steps I found useful.

installing gnome and most of the distro (including tcsh as it is my favorite shell)

- change nautilus to open always in browser
- remove SElinux policy
- remove option in keyboard options - allow mouse to be controlled by keypad
- add vino vnc option to user
- change autologin for user: vi /etc/gdm/gdm.schemas
- add user to sudo:

su --login -c 'visudo'
Now go below the line:
root    ALL=(ALL)       ALL
press a, and type the following
username ALL=(ALL)      ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel        ALL=(ALL)       ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel        ALL=(ALL)       NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.

- adding windows fonts:
  wget http://www.my-guides.net/en/images/stories/fedora10/msttcore-fonts-2.0-2.noarch.rpm
  su -c 'rpm -ivh msttcore-fonts-2.0-2.noarch.rpm'
- open terminal here in nautilus
  su -c 'yum install nautilus-open-terminal'
- Enable Num Lock on GNOME startup
  su -c 'yum install numlockx'
  su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
  su -c 'gedit /etc/gdm/Init/Default'
  And add the following lines at the end of the file before the exit 0.
  if [ -x /usr/bin/numlockx ]; then
    /usr/bin/numlockx on
  fi

- add to install:
  yum extender
  yum Fastest Mirror Plugin
  GMT
  qgis
  grace
  filezilla
  ipython
  KDocker
  google-earth
  wine
  g77
  bluecurve icons
 gnash - for flash viewing

- setting /etc/X11/xorg.conf for display with no monitor attached:
make sure to use a video card which is not onboard (or perhaps this is only valid for the 965 intel mobo?)

- edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]

- adding python modules
for gui: http://easygui.sourceforge.net/index.html
scipy
numpy
matplotlib
ipython
pylab

in a case of emergency: enabling write on a system error:
mount -t ext3 /dev/sda1 / -o remount,rw

- installing ROI PAC
- adding axillary data

Working on two machines at once

The Problem:
On my work I use two computers, one is Linux based which carry out all the heavy processing and another is windows based for viewing results, office related jobs and other Microsoft compatibility issues.

My problem was I had two mice, two keyboard and two screens. it is very confusing and tiring to switch from one to another.

The Solution:
The Linux distro comes with a wonderful component - VINO SERVER a VNC server which connects to your display 0 (the main display).
so I installed REALVNC viewer on my windows computer, connected one screen to the DVI exit and one to the D-sub (VGA) exit on the graphic card.
now, all need to do is to enter the Linux machine via VNC, drag the window to the second screen and use "Full Screen" view. This allows me to use one set of keyboard and mouse and move from one machine to another just by moving from one screen to another.

Tips:

• autologin:
  Since you can't "see" the Linux machine without connecting to it, you can tweak the bootloader to load you user automatically:
  edit /etc/gdm/custom.conf (as superuser if needed) and change:
  [daemon]
  TimedLoginEnable=true
  TimedLogin=yourusername
  TimedLoginDelay=0
• ssh first:
  if you don't want to use autologin for security or other reason, you can login via ssh and start a gnome session and then use the VNC viewer. just enable ssh daemon and vino server.
• othere VNC servers:
  you can always use different VNC server and connect to other displays then 0.
• graphic card:
  I tried to use this method with an onboard G. card and only got low resolution screen size.
  it might be my mother board (intel DQ965GF) or bad X settings but using a GeForce card I managed to use my desired resolution with no problems.

Welcom to My Blog.

My name is Ran Novitsky Nof. I currently work at the Geophysical Institute of Israel.

This blog's first aim is to serve as a holding place for my ideas, thoughts, tips and workflows during my research and sharing it with the world - if only one person would benefit from something here - I did my share.

Additional Info may be found at my personal homepages : ran.rnof.info