After installing Fedora 17 I will now describe how to install MySQL server and SeisComp3.
first, install MySQL (following this):
> sudo yum install -y mysql mysql-server
enable and start MySQL service:
> sudo systemctl enable mysqld.service
> sudo systemctl start mysqld.service
Next, secure the server:
> sudo /usr/bin/mysql_secure_installation
(answer yes to all questions and change the MySQL root password)
If you need to change the database location from the default follow this (source):
Stop the service:
> systemctl stop mysqld.service
Copy from default to new location:
> cp -R -p /var/lib/mysql /path/to/new/datadir
Clean unneeded files:
> rm /path/to/new/datadir
(don't worry about error messages - the undeleted files are the databases)
Edit the configuration file:
> gedit /etc/mysql/my.cnf
(change keyword value of datadir to the new location)
Restart the service:
> systemctl restart mysqld.service
Now to the SeisComp3 install (Zurich version for fedora16 modified for fedora 17):
Add needed packages:
> sudo yum install -y boost-filesystem boost-iostreams boost-program-options boost-serialization boost-signals
boost-thread boost-wave boost-regex boost-devel ncurses python-numeric numpy alsa-utils festival
Link installed libraries (so.1.48) to the needed packages (so.1.47):
> sudo ln -s /usr/lib64/libboost_iostreams.so.1.48.0 /usr/lib64/libboost_iostreams.so.1.47.0
> sudo ln -s /usr/lib64/libboost_filesystem.so.1.48.0 /usr/lib64/libboost_filesystem.so.1.47.0
> sudo ln -s /usr/lib64/libboost_system.so.1.48.0 /usr/lib64/libboost_system.so.1.47.0
> sudo ln -s /usr/lib64/libboost_regex.so.1.48.0 /usr/lib64/libboost_regex.so.1.47.0
> sudo ln -s /usr/lib64/libboost_thread-mt.so.1.48.0 /usr/lib64/libboost_thread-mt.so.1.47.0
> sudo ln -s /usr/lib64/libboost_program_options.so.1.48.0 /usr/lib64/libboost_program_options.so.1.47.0
> sudo ln -s /usr/lib64/libboost_signals.so.1.48.0 /usr/lib64/libboost_signals.so.1.47.0
Put key files in ~/.seiscomp3/key/ directory
Install using the setup and seiscomp config procedures described in the SeisComp3 documentations
add a source to env.sh file in your .bashrc:
source ~/seiscomp3/lib/env.sh
For the seattle release use:
>sudo yum install -y boost boost-devel python-numeric
Don't forget to download and extract to the seiscomp directory, the correct fedora seiscomp package (e.g. seiscomp3-zurich-xxxx.xxx.xx-fedora16-x86_64.tar.gz), the maps (e.g. seiscomp3-maps.tar.gz) and the methadata of GE network (e.g. seiscomp3-metadata-GE-2010.tar.gz).
Monday, September 24, 2012
Tuesday, August 28, 2012
Howto write to stdout XY
how to use python to write to a certain location (x,y) on the stdout (terminal):
copied form here
import sys
def print_there(x, y, text):
sys.stdout.write("\x1b7\x1b[%d;%df%s\x1b8" % (x, y, text))
sys.stdout.flush()
Creating a software raid
1. format the hard drives to raid using fdisk (gdisk in fedora 17)
or with parted (for 3TB disks):
> parted /dev/sdx
(parted) mklabel gpt
(parted) unit TB
(parted) mkpart primary 0 3001G
(parted) print
(parted) quit
2. Create the raid device (change level and devices as needed):
mdadm --create --level=1 --raid-devices=2 /dev/md0 /dev/sd[bc]1
3. format the file system of the new device:
mkfs.ext4 /dev/md0
4. create the mdadm.conf file:
mdadm --detail --scan > /etc/mdadm.conf
5. add mail alert:
echo "MAILADDR your_mail@somthing.com" >> /etc/mdadm.conf
echo "MAILFROM your_server" >> /etc/mdadm.conf
6. test mail alert:
mdadm --monitor --scan --test --oneshot
7. start mail alert deamon:
systemctl enable mdmonitor.service
systemctl start mdmonitor.service
8. edit the mail aliases file to forward root messages to your email:
vi /etc/aliases
add your mail under the line: # Person who should get root's mail
http://www.iceteks.com/articles.php/linuxmdadmraid/
* after name change, create the /etc/mdadm.conf file and run:
sudo update-initramfs -u
or
sudo dracut -f /boot/initramfs-[current].img
or with parted (for 3TB disks):
> parted /dev/sdx
(parted) mklabel gpt
(parted) unit TB
(parted) mkpart primary 0 3001G
(parted) print
(parted) quit
2. Create the raid device (change level and devices as needed):
mdadm --create --level=1 --raid-devices=2 /dev/md0 /dev/sd[bc]1
3. format the file system of the new device:
mkfs.ext4 /dev/md0
4. create the mdadm.conf file:
mdadm --detail --scan > /etc/mdadm.conf
5. add mail alert:
echo "MAILADDR your_mail@somthing.com" >> /etc/mdadm.conf
echo "MAILFROM your_server" >> /etc/mdadm.conf
6. test mail alert:
mdadm --monitor --scan --test --oneshot
7. start mail alert deamon:
systemctl enable mdmonitor.service
systemctl start mdmonitor.service
8. edit the mail aliases file to forward root messages to your email:
vi /etc/aliases
add your mail under the line: # Person who should get root's mail
run newaliases command
see also at:http://www.iceteks.com/articles.php/linuxmdadmraid/
* after name change, create the /etc/mdadm.conf file and run:
sudo update-initramfs -u
or
sudo dracut -f /boot/initramfs-[current].img
Monday, August 27, 2012
Installing Fedora Core 17
based on my-guides post-install post
1. Add to sudo:
> su --login -c 'visudo'
add under the line:
root ALL=(ALL) ALL
the line:
username ALL=(ALL) ALL
and remove the sign # from the line with %wheel
save and test using the line:
> sudo whoami
(answer should be root)
2. Disable SeLinux and firewall:
> sudo vi /etc/selinux/config
change enabled to disabled
> sudo systemctl disable iptables.service
> sudo systemctl stop iptables.service
3. change autologin for user:
sudo vi /etc/gdm/custom.conf
under [daemon], add:
TimedLoginEnable=true
TimedLogin=yourUserName
TimedLoginDelay=1
4. Switch to gnome3 fallback mode:
> gsettings set org.gnome.desktop.session session-name 'gnome-fallback'
5. enable sshd:
>sudo systemctl enable sshd.service
> sudo systemctl start sshd.service
6. auto rotate screen on login:
add to .profile on home:
xrandr --output VGA-1 --rotate left
7.add daily update report (as root):
following this:
> vi /etc/cron.daily/yum_update_note
#!/bin/sh
if [ ! -f /var/lock/subsys/yum ] ; then
msg=`/usr/bin/yum check-update 2>&1`
i=`echo "$msg" | wc -l`
if ((i>1)); then
echo "$msg" | /bin/mail -s "yum check-update output" root
fi
fi
and chnage mode for executable:
> sudo chmod +x /etc/cron.daily/yum_update_note
1. Add to sudo:
> su --login -c 'visudo'
add under the line:
root ALL=(ALL) ALL
the line:
username ALL=(ALL) ALL
and remove the sign # from the line with %wheel
save and test using the line:
> sudo whoami
(answer should be root)
2. Disable SeLinux and firewall:
> sudo vi /etc/selinux/config
change enabled to disabled
> sudo systemctl disable iptables.service
> sudo systemctl stop iptables.service
3. change autologin for user:
sudo vi /etc/gdm/custom.conf
under [daemon], add:
TimedLoginEnable=true
TimedLogin=yourUserName
TimedLoginDelay=1
4. Switch to gnome3 fallback mode:
> gsettings set org.gnome.desktop.session session-name 'gnome-fallback'
5. enable sshd:
>sudo systemctl enable sshd.service
> sudo systemctl start sshd.service
6. auto rotate screen on login:
add to .profile on home:
xrandr --output VGA-1 --rotate left
7.add daily update report (as root):
following this:
> vi /etc/cron.daily/yum_update_note
#!/bin/sh
if [ ! -f /var/lock/subsys/yum ] ; then
msg=`/usr/bin/yum check-update 2>&1`
i=`echo "$msg" | wc -l`
if ((i>1)); then
echo "$msg" | /bin/mail -s "yum check-update output" root
fi
fi
and chnage mode for executable:
> sudo chmod +x /etc/cron.daily/yum_update_note
Tuesday, May 29, 2012
Thursday, May 10, 2012
Howto run a process automatically when a certain mail is downloaded by Thunderbird
In order to run a file whenever a certain mail arrives to my inbox I created a filter using the Thunderbird add-on FiltaQuilla. The add-on expand the messages filtering options and add an option to "run a file". when a certain mail match the filter, Thunderbird will run the file specified. I use a python code - code.py so in the filter configuration under Linux (Fedora core 14) I use this line:
/usr/bin/python,/home/UserName/code.py,-a
The line is a comma separated 3 parts line. The first part is the python interperter, the second is the script name to run and the last part is an argument needed for the script to run. Omitting the first part and the filter wont work.
/usr/bin/python,/home/UserName/code.py,-a
The line is a comma separated 3 parts line. The first part is the python interperter, the second is the script name to run and the last part is an argument needed for the script to run. Omitting the first part and the filter wont work.
Monday, November 21, 2011
HowTo Mosaic ASTER Global Digital Elevation Model (GDEM)
A second version of ASTER GDEM was released in Oct. 17 2011 by The Ministry of Economy, Trade and Industry of Japan (METI) and the National Aeronautics and Space Administration (NASA). The GDEM is in a 30m pixel resolution and 1x1 degree tiles. The data is distributed as zipped GeoTIFF files. When a larger than 1x1 deg. DEM is needed for Interferometry processing, you can use my GDEM.py script available from my script page. This python script will need a system with pylab and PIL python modules.
Simply:
Download the GDEM zipped files (see here on how to do that).
Unzip the zip files in the working directory.
The working directory should contain sub-directories containing the GeoTIFF files.
Run the script giving -h for help or the final DEM name as arguments.
The Final DEM is aimed for using as DEM in ROI-PAC interferometry processing.
If a -g is added as an argument, the produced DEM will be suitable for interferometric processing using Gamma software.
Saturday, July 30, 2011
Create a photo collage using a single command-line
In order to create a contact-sheet or a collage of many files under linux, I prefer to use a single command-line. I create a directory with all the images and use ImageMagick montage function:
montage -font Arial-Narrow -pointsize 10 -label '%t' -resize 100x150 *.jpg -geometry +3+3 -tile 4x4 -shadow collage.jpg
Thursday, July 28, 2011
HowTo use a computer without a screen attached
The Problem:
I have a Linux machine (Fedora 14) with no screen attached. I view the graphics via VNC server. I had to adjust the system to use the Nvidia driver and tweak it for no screen.
The solution:
First we need to install the Nvidia driver. Then, we need to adjust the X configuration for no screen and our preferred resolution.
for more details see: http://forums.fedoraforum.org/showthread.php?t=204752
1. Update the kernel:
2. Add rpmfusion to your repository.
3. Depending on your card add nvidia akmod to the kernel and the proper driver. for a Gforce 9500GT on my 64bit machine:
4. Edit the xorg.conf file. don't forget to adjust the Nvidia card for no screen attached by adding:
Option "ConnectedMonitor" "CRT"
to the device section and adjust the screen resolution under the Screen section:
I have a Linux machine (Fedora 14) with no screen attached. I view the graphics via VNC server. I had to adjust the system to use the Nvidia driver and tweak it for no screen.
The solution:
First we need to install the Nvidia driver. Then, we need to adjust the X configuration for no screen and our preferred resolution.
for more details see: http://forums.fedoraforum.org/showthread.php?t=204752
1. Update the kernel:
su yum update kernel* reboot
2. Add rpmfusion to your repository.
su rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm
3. Depending on your card add nvidia akmod to the kernel and the proper driver. for a Gforce 9500GT on my 64bit machine:
yum install akmod-nvidia-260.19.36-1.fc14.x86_64 xorg-x11-drv-nvidia-libs-260.19.36-1.fc14.x86_64
4. Edit the xorg.conf file. don't forget to adjust the Nvidia card for no screen attached by adding:
Option "ConnectedMonitor" "CRT"
to the device section and adjust the screen resolution under the Screen section:
Section "Device"
Identifier "Card0"
Driver "nvidia"
Option "ConnectedMonitor" "CRT"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "Monitor0"
SubSection "Display"
Viewport 0 0
Modes "1280x1024"
EndSubSection
EndSection
Thursday, July 14, 2011
Daichi - ALOS : finding Images and Baseline Calculation
Using JAXA satellite "DAICHI" (ALOS) for SAR Interferometry might raise the need for calculating perpendicular baselines. One can use AUIG site to locate the needed images for interferometry and use PALSAR baseline calculator CalcBP for calculating perpendicular baselines. more info is available here.
Thursday, March 31, 2011
Installing Linux Fedora Core 14
Well, after a long time using Fedora 11 I'm moving on to Fedora 14.
Here I'll detail the steps I used for the installation.
I generally follow my Fedora 10 installation guide with some changes of course.
LiveCD:
I use a liveCD (download page) to install.
after loading the system via cd,
I use the "Install to Hard Drive" icon.
next,next,next.
Giving the Hostname, setting location, giving a good root password.
File System:
Since I use several hard drives I need to configure the file system. I use one 1.5TB disk as my home, another one for data and processing and one for the system so all the installation will only deal with the last one. later I'll mount all the rest of the disks.
so the type of installation for me is "Replace Existing Linux System(s)", but with "Review and modify layout" switch on and I select the system disk as "Install Target Devices".
Next I change the partitions and mount points of all disks as needed.
Finally, The boot loader is set up and the system is installed on the hard drive.
Updates and softwares:
As a first step few softwares should be installed and update (this might take forever):
Customizing:
remove SElinux policy
disable filrewall
configure remote desktop
add user to sudo:
su --login -c 'visudo'
Now go below the line:
root ALL=(ALL) ALL
press a, and type the following
username ALL=(ALL) ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.
change autologin for user:
sudo vi /etc/gdm/custom.conf
under [daemon], add:
AutomaticLoginEnable=true
AutomaticLogin=yourUserName
edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]
Enable Num Lock on GNOME startup:
su -c 'yum install numlockx'
su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
su -c 'gedit /etc/gdm/Init/Default'
And add the following lines at the end of the file before the exit 0.
if [ -x /usr/bin/numlockx ]; then
/usr/bin/numlockx on
fi
set the samba server
set the mail server:
sudo vi /etc/ssmtp/ssmtp.conf
change the line:
mailhub=mail
to:
mailhub=[your outgoing mail server]
Set the screen resolution for vnc and no monitor:
create xorg.conf file
sudo Xorg :1 -configure
sudo cp /root/xorg.conf.new /etc/X11/xorg.conf
and add lines to the xorg.conf file:
Section "Monitor"
Identifier "Monitor0"
ModelName "LCD Panel 1280x1024"
HorizSync 31.5 - 64.0
VertRefresh 56.0 - 65.0
Option "dpms"
EndSection
Section "Device"
### Available Driver options are:-
### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
### <percent>: "<f>%"
### [arg]: arg optional
#Option "SWcursor" # [<bool>]
#Option "HWcursor" # [<bool>]
#Option "NoAccel" # [<bool>]
#Option "ShadowFB" # [<bool>]
#Option "UseFBDev" # [<bool>]
#Option "Rotate" # [<str>]
#Option "VideoKey" # <i>
#Option "FlatPanel" # [<bool>]
#Option "FPDither" # [<bool>]
#Option "CrtcNumber" # <i>
#Option "FPScale" # [<bool>]
#Option "FPTweak" # <i>
#Option "DualHead" # [<bool>]
Identifier "Card0"
Driver "vesa"
BusID "PCI:1:0:0"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "Monitor0"
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1280x1024"
EndSubSection
EndSection
set the panels and desktop background as you like it.
Install RPMfusion repositories following: www.rpmfusion.org/Configuration
add true type fonts from windows
Here I'll detail the steps I used for the installation.
I generally follow my Fedora 10 installation guide with some changes of course.
LiveCD:
I use a liveCD (download page) to install.
after loading the system via cd,
I use the "Install to Hard Drive" icon.
next,next,next.
Giving the Hostname, setting location, giving a good root password.
File System:
Since I use several hard drives I need to configure the file system. I use one 1.5TB disk as my home, another one for data and processing and one for the system so all the installation will only deal with the last one. later I'll mount all the rest of the disks.
so the type of installation for me is "Replace Existing Linux System(s)", but with "Review and modify layout" switch on and I select the system disk as "Install Target Devices".
Next I change the partitions and mount points of all disks as needed.
Finally, The boot loader is set up and the system is installed on the hard drive.
Updates and softwares:
As a first step few softwares should be installed and update (this might take forever):
system-config-network
yumex
gconf-editor
gedit-plugins
gthumb
nautilus0open-terminal
tango-icon-theme
tango-icon-extras
fonts for Hebrew
all stix fonts
cm-lgc fonts
blender
gimp
gv
ImageMagick
inkscape
filezilla
thunderbird
emacs
open office
vim-enhanced
gcc
gmt
grace
HDF
HDF5
lapack
netcdf
numpy
python support for open office
compat-gcc
glade3
ipython
pango-devel
wget
tcsh
system-config-samba
system-config-users
system-config-nfs
dos2unix
selinux configuration gui
matplotlib
scipy
scitools
kdocker
numlockx
bluecurve icons
ssmtp
yumex
gconf-editor
gedit-plugins
gthumb
nautilus0open-terminal
tango-icon-theme
tango-icon-extras
fonts for Hebrew
all stix fonts
cm-lgc fonts
blender
gimp
gv
ImageMagick
inkscape
filezilla
thunderbird
emacs
open office
vim-enhanced
gcc
gmt
grace
HDF
HDF5
lapack
netcdf
numpy
python support for open office
compat-gcc
glade3
ipython
pango-devel
wget
tcsh
system-config-samba
system-config-users
system-config-nfs
dos2unix
selinux configuration gui
matplotlib
scipy
scitools
kdocker
numlockx
bluecurve icons
ssmtp
Customizing:
remove SElinux policy
disable filrewall
configure remote desktop
add user to sudo:
su --login -c 'visudo'
Now go below the line:
root ALL=(ALL) ALL
press a, and type the following
username ALL=(ALL) ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.
change autologin for user:
sudo vi /etc/gdm/custom.conf
under [daemon], add:
AutomaticLoginEnable=true
AutomaticLogin=yourUserName
edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]
Enable Num Lock on GNOME startup:
su -c 'yum install numlockx'
su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
su -c 'gedit /etc/gdm/Init/Default'
And add the following lines at the end of the file before the exit 0.
if [ -x /usr/bin/numlockx ]; then
/usr/bin/numlockx on
fi
set the samba server
set the mail server:
sudo vi /etc/ssmtp/ssmtp.conf
change the line:
mailhub=mail
to:
mailhub=[your outgoing mail server]
Set the screen resolution for vnc and no monitor:
create xorg.conf file
sudo Xorg :1 -configure
sudo cp /root/xorg.conf.new /etc/X11/xorg.conf
and add lines to the xorg.conf file:
Section "Monitor"
Identifier "Monitor0"
ModelName "LCD Panel 1280x1024"
HorizSync 31.5 - 64.0
VertRefresh 56.0 - 65.0
Option "dpms"
EndSection
Section "Device"
### Available Driver options are:-
### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
### <percent>: "<f>%"
### [arg]: arg optional
#Option "SWcursor" # [<bool>]
#Option "HWcursor" # [<bool>]
#Option "NoAccel" # [<bool>]
#Option "ShadowFB" # [<bool>]
#Option "UseFBDev" # [<bool>]
#Option "Rotate" # [<str>]
#Option "VideoKey" # <i>
#Option "FlatPanel" # [<bool>]
#Option "FPDither" # [<bool>]
#Option "CrtcNumber" # <i>
#Option "FPScale" # [<bool>]
#Option "FPTweak" # <i>
#Option "DualHead" # [<bool>]
Identifier "Card0"
Driver "vesa"
BusID "PCI:1:0:0"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Card0"
Monitor "Monitor0"
SubSection "Display"
Viewport 0 0
Depth 24
Modes "1280x1024"
EndSubSection
EndSection
set the panels and desktop background as you like it.
Install RPMfusion repositories following: www.rpmfusion.org/Configuration
add true type fonts from windows
Wednesday, October 13, 2010
RoiView - Explore InSAR data and more
RoiView was originaly designed to replace DGX/MDX software which require bureaucracy to obtain. This software was used in order to view ROI-PAC interferometry processing results but producing an image from file was not so strait forward. In time support was added for ERDAS ER Mapper header files and for user provided parameters with or without header files.
Current RoiView version (0.75) is available for download at
See also RoiView's help files
Screenshot:
Current RoiView version (0.75) is available for download at
See also RoiView's help files
Screenshot:
Tuesday, September 28, 2010
HowTo Parse HTML text using python.
I needed to to get the text of a specific div elemnt in an html file. I tried to use python's standard library modules for markup processing such as htmllib etc. but I couldn't figure how to use them. I've created my own module just for getting the text from an html element:
#!/bin/env python
from htmlentitydefs import entitydefs as ent
import string
# This module enable you to extract text from a certian HTML element
#
# by Ran Novitsky Nof, 2010
# ran.nof@gmail.com
#
# example of use:
# say we want to get the text in an element of type tag (e.g. 'div','a','span' etc.)
# who has an attribute key (e.g. "id","class","href" etc.) with a value of val
# for example in order to extract the text of a div element with id of textdiv from a file htmlfile.html:
#
# <html>
# <head>
# :
# </head>
# <body>
# :
# <div "id"="textdiv">I will not buy this <a href="spam">record</a> it is scratched. </div>
# :
# </body>
# </html>
#
# use:
# from htmlparser import Parser
# htmlfile='htmlfile.html'
# tag,key,val = ('div','id','textdiv')
# text=Parser(htmlfile).getText(tag,key,val)
# print(text)
#
class Element():
def __init__(self):
self.startTag = -1
self.endTag = -1
self.attrib = {}
self.keys = self.attrib.keys()
self.innerHTML = ''
self.tag = ''
self.start = -1
self.end = -1
class Parser():
def __init__(self,infile):
self._root = open(infile).read()
self._root = self._root[self._root.find('<body'):self._root.find('</body')]
self._root = self._root[self._root.find('>')+1:]
self.tags = set()
i,j=0,0
self.tagstarts = {}
self.tagends = {}
self.elements = []
while i<len(self._root):
i = self._root.find('<',i)
j = self._root.find('>',i)
if not j>i: break
tag = self._root[i+1:j].split()[0]
if tag.startswith('/'):
tag = tag[1:]
self.tagends[tag][-1-self.tagends[tag][::-1].index(None)]=((i,j))
else:
self.tags.add(tag)
if tag in self.tagstarts:
self.tagstarts[tag].append((i,j))
self.tagends[tag].append(None)
else:
self.tagstarts[tag]=[(i,j)]
self.tagends[tag]=[None]
i=j+1
self.getElements()
def getElements(self):
for tag in self.tags:
if not tag in ['img']:
for i in range(len(self.tagstarts[tag])):
element = Element()
element.startTag = self.tagstarts[tag][i][0]
element.endTag = self.tagstarts[tag][i][1]
tagData = self._root[element.startTag+1:element.endTag].replace("\"","")
element.tag = tag
element.attrib=dict([a.split('=') for a in tagData.split() if '=' in a])
element.start = element.startTag
element.end = self.tagends[tag][i][1]
element.innerHTML = (self.tagstarts[tag][i][1]+1,self.tagends[tag][i][0])
self.elements.append(element)
def getText(self,tag,key,val):
element = [element for element in self.elements if element.tag==tag and element.attrib[key]==val]
if len(element):
element = element[0]
start,end = element.innerHTML
text = self._root[start:end]
i,j=0,0
while i<len(text):
i = text.find('<',i)
j = text.find('>',i)
if i<0 or j<0: break
text = text[:i]+text[j+1:]
i,j=-1,-1
while i<len(text):
i = text.find('&',i+1)
j = text.find(';',i+1)
if i<0 or j<0: break
if text[i+1:j] in ent.keys(): text = text[:i]+ent[text[i+1:j]]+text[j+1:]
else:
text=None
return text
Note the code does not check if the html code is correct. it also work only for the body part, and ignore img tags.from htmlentitydefs import entitydefs as ent
import string
# This module enable you to extract text from a certian HTML element
#
# by Ran Novitsky Nof, 2010
# ran.nof@gmail.com
#
# example of use:
# say we want to get the text in an element of type tag (e.g. 'div','a','span' etc.)
# who has an attribute key (e.g. "id","class","href" etc.) with a value of val
# for example in order to extract the text of a div element with id of textdiv from a file htmlfile.html:
#
# <html>
# <head>
# :
# </head>
# <body>
# :
# <div "id"="textdiv">I will not buy this <a href="spam">record</a> it is scratched. </div>
# :
# </body>
# </html>
#
# use:
# from htmlparser import Parser
# htmlfile='htmlfile.html'
# tag,key,val = ('div','id','textdiv')
# text=Parser(htmlfile).getText(tag,key,val)
# print(text)
#
class Element():
def __init__(self):
self.startTag = -1
self.endTag = -1
self.attrib = {}
self.keys = self.attrib.keys()
self.innerHTML = ''
self.tag = ''
self.start = -1
self.end = -1
class Parser():
def __init__(self,infile):
self._root = open(infile).read()
self._root = self._root[self._root.find('<body'):self._root.find('</body')]
self._root = self._root[self._root.find('>')+1:]
self.tags = set()
i,j=0,0
self.tagstarts = {}
self.tagends = {}
self.elements = []
while i<len(self._root):
i = self._root.find('<',i)
j = self._root.find('>',i)
if not j>i: break
tag = self._root[i+1:j].split()[0]
if tag.startswith('/'):
tag = tag[1:]
self.tagends[tag][-1-self.tagends[tag][::-1].index(None)]=((i,j))
else:
self.tags.add(tag)
if tag in self.tagstarts:
self.tagstarts[tag].append((i,j))
self.tagends[tag].append(None)
else:
self.tagstarts[tag]=[(i,j)]
self.tagends[tag]=[None]
i=j+1
self.getElements()
def getElements(self):
for tag in self.tags:
if not tag in ['img']:
for i in range(len(self.tagstarts[tag])):
element = Element()
element.startTag = self.tagstarts[tag][i][0]
element.endTag = self.tagstarts[tag][i][1]
tagData = self._root[element.startTag+1:element.endTag].replace("\"","")
element.tag = tag
element.attrib=dict([a.split('=') for a in tagData.split() if '=' in a])
element.start = element.startTag
element.end = self.tagends[tag][i][1]
element.innerHTML = (self.tagstarts[tag][i][1]+1,self.tagends[tag][i][0])
self.elements.append(element)
def getText(self,tag,key,val):
element = [element for element in self.elements if element.tag==tag and element.attrib[key]==val]
if len(element):
element = element[0]
start,end = element.innerHTML
text = self._root[start:end]
i,j=0,0
while i<len(text):
i = text.find('<',i)
j = text.find('>',i)
if i<0 or j<0: break
text = text[:i]+text[j+1:]
i,j=-1,-1
while i<len(text):
i = text.find('&',i+1)
j = text.find(';',i+1)
if i<0 or j<0: break
if text[i+1:j] in ent.keys(): text = text[:i]+ent[text[i+1:j]]+text[j+1:]
else:
text=None
return text
Sunday, July 18, 2010
SSH with no password
When you wish to login to a remote system through an ssh session without using any password you'll need to produce a key on the local computer and on the remote computer in order to allow password - free entrance.
first you'll need to produce the key in the local computer:
ssh-keygen -t dsa -b 1024 -P ""
press Enter on the questions followed by the command.
the output will be something like:
Your identification has been saved in /home/yourname/.ssh/id_dsa.
Your public key has been saved in /home/yourname/.ssh/id_dsa.pub.
The key fingerprint is:
4f:9e:4c:17:8f:19:37:32:bc:45:4f:2a:a2:8e:1c:15 yourname@your.loacl.machine
The key's randomart image is:
+--[ DSA 1024]----+
| .o |
| .oo. |
| . o+ |
| o *.+. .|
| .E . + . |
| o .+.. . .|
| +o.o. + |
| o+E |
| o=. |
+-----------------+
first you'll need to produce the key in the local computer:
ssh-keygen -t dsa -b 1024 -P ""
press Enter on the questions followed by the command.
the output will be something like:
Your identification has been saved in /home/yourname/.ssh/id_dsa.
Your public key has been saved in /home/yourname/.ssh/id_dsa.pub.
The key fingerprint is:
4f:9e:4c:17:8f:19:37:32:bc:45:4f:2a:a2:8e:1c:15 yourname@your.loacl.machine
The key's randomart image is:
+--[ DSA 1024]----+
| .o |
| .oo. |
| . o+ |
| o *.+. .|
| .E . + . |
| o .+.. . .|
| +o.o. + |
| o+E |
| o=. |
+-----------------+
This will produce two files in you .ssh directory: id_dsa and id_dsa.pub
cat id_dsa.pub into ~/.ssh/authorized_keys on the remote system you want to log into.
change mod to 600:
chmod 600 ~/.ssh/authorized_keys
(source of the tip)
change mod to 600:
chmod 600 ~/.ssh/authorized_keys
Friday, June 25, 2010
Howto export points to a Google Earth KML file
The problem:
A table of xyz data points can be viewed spatially with different methods:
GMT and ArcView are just a couple of options. I often get tables of spatial data points in Microsoft Excel (xp ver.) format or CSV of some kind. An easy (and free) way to visualize the data points on Windows would be to export the data and view it on Google Earth.
The solution:
points2kml (115kb).
In order to ease the work-flow I wrote a VBA macro add-in for Excel that exports selected fields (longitude, latitude, name and data) to a Google Earth KML file. the point2kml add-in and can be found on my scripts page.
A table of xyz data points can be viewed spatially with different methods:
GMT and ArcView are just a couple of options. I often get tables of spatial data points in Microsoft Excel (xp ver.) format or CSV of some kind. An easy (and free) way to visualize the data points on Windows would be to export the data and view it on Google Earth.
The solution:
points2kml (115kb).
In order to ease the work-flow I wrote a VBA macro add-in for Excel that exports selected fields (longitude, latitude, name and data) to a Google Earth KML file. the point2kml add-in and can be found on my scripts page.
Wednesday, June 2, 2010
How to format a hard drive on Linux
The Problem:
How to format a hard drive on Linux.
The Solution:
in order to format a HD on Linux you'll need super user privileges.
you can use sudo if you have sudo privilages or in the terminal
switch to super user:
>su
(give super user pwd)
and ignore the sudo in the following commands. if you're system can't find the commands while in su mode, try to add /sbin/ before the command.
follow the steps below: marked with •
run commnads marked with >
commands output looks like this.
Warning: formatting your HD will erase all data on it!
Now lets say our HD is sdb1.
Command (m for help):
> sudo mkfs -t ext2 /dev/hdb1
remember to change the number at the end if your HD device number is different.
if your using different fs than ext2 (newer Linux systems have ext3 and fc11 has ext4) replace the ext2 in the command with the correct fc name.
the mkfs may take some time to run (depending on the computer and HD size).
To add a label to the partition use:
e2label /dev/sdb1 NEWLABEL
where sdb should be changed to the correct device and NEWLABEL changed to the new label.
Now add the device to the fstab file in order to mount it:
> sudo vi /etc/fstab
and add a line like:
LABEL=NEWLABEL MOUNTPOINT ext3 defaults 1 2
where NEWLABEL is the disk labe and MOUNTPOINT is where you want to mount the disk.
mount it using:
> sudo mount -a
Note you might need to change permissions and owner of the mount point:
> sudo chown username:usergroop MOUNTPOINT
when finnished - reboot.
How to format a hard drive on Linux.
The Solution:
in order to format a HD on Linux you'll need super user privileges.
you can use sudo if you have sudo privilages or in the terminal
switch to super user:
>su
(give super user pwd)
and ignore the sudo in the following commands. if you're system can't find the commands while in su mode, try to add /sbin/ before the command.
follow the steps below: marked with •
run commnads marked with >
commands output looks like this.
Warning: formatting your HD will erase all data on it!
- Connect the HD to the computer, make sure it's on.
- get the hard drive's (device) name on your system:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_insarlablin-lv_root
908G 77G 786G 9% /
/dev/sda1 194M 31M 153M 17% /boot
/dev/sdb1 1.4T 1.1T 193G 86% /home/novitsky
/dev/sdc1 1.4T 1.1T 193G 86% /home/novitsky/data
Now lets say our HD is sdb1.
- lets build it's partition:
The number of cylinders for this disk is set to 182401.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help):
- pressing m will get this menu:
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)
- pressing p will print the current partition:
Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000923e6
Device Boot Start End Blocks Id System
/dev/sdb1 1 182401 1465136001 83 Linux
- pressing d will delete the current partition. depending on the partition you want to delete (this case 1) press the number of the partition to be deleted. if all the disk should be formatted and there are several partitions repeat pressing d and the number of the partitions until you delete them all.
- pressing n will create a new partition.
- pressing p will create a primary partition. select 1 as the number of the partition. use the defaults (just press Enter) for the start and end cylinders.
- pressing t will set the file system type.
- pressing l will list the available file systems.
- select 83 if you want a Linux file system
- pressing w will write and exit to shell.
> sudo mkfs -t ext2 /dev/hdb1
remember to change the number at the end if your HD device number is different.
if your using different fs than ext2 (newer Linux systems have ext3 and fc11 has ext4) replace the ext2 in the command with the correct fc name.
the mkfs may take some time to run (depending on the computer and HD size).
To add a label to the partition use:
e2label /dev/sdb1 NEWLABEL
where sdb should be changed to the correct device and NEWLABEL changed to the new label.
Now add the device to the fstab file in order to mount it:
> sudo vi /etc/fstab
and add a line like:
LABEL=NEWLABEL MOUNTPOINT ext3 defaults 1 2
where NEWLABEL is the disk labe and MOUNTPOINT is where you want to mount the disk.
mount it using:
> sudo mount -a
Note you might need to change permissions and owner of the mount point:
> sudo chown username:usergroop MOUNTPOINT
when finnished - reboot.
Wednesday, April 28, 2010
Moving memory from swap (page) to RAM
When using more memory than your RAM Linux will use swap which is placed on your hard drive and is much slower. Sometimes after using swap there might be a situation where the RAM is free but some of your data is on the swap. There is a way to relocate the swap data to the RAM.
more details and a nice script to do is are available at: Ubuntu Community Documentation Swap FAQ page.
The commands simply shut off swap and turning it on back again.
You'll have to make sure there is enough space on the RAM to accommodate the Swap data, this can be done using:
the output will show you the used space of Swap and free space of RAM:
The example above show that no data (0) is stored on Swap.
more details and a nice script to do is are available at: Ubuntu Community Documentation Swap FAQ page.
sudo swapoff -a
sudo swapon -a
The commands simply shut off swap and turning it on back again.
You'll have to make sure there is enough space on the RAM to accommodate the Swap data, this can be done using:
free
the output will show you the used space of Swap and free space of RAM:
total used free shared buffers cached Mem: 8047896 772268 7275628 0 19540 129372 -/+ buffers/cache: 623356 7424540 Swap: 10108920 0 10108920
The example above show that no data (0) is stored on Swap.
Wednesday, April 21, 2010
Using hillshade image as intensity (improved matplotlib shade)
Matplotlib module enables hillshade method (shade) using a LightSource class (v 0.99).
The problem is it uses the data itself as intensity and data. It is very useful for viewing a DEM but sometimes you would like the DEM as intensity underlying some other data. Another problem is that the shade method is producing a very light colored image sometimes even white where intensity is high.
I used as an example a DEM derived from SRTM v4 data acquired at the International Centre for Tropical Agriculture (CIAT - http://srtm.csi.cgiar.org) the hillshade production was made using LightSource class with azimuth of 165 deg. and altitude of 45 deg.)
The difference in the shading colors derived from the method used to produce it. While the matplotlib method uses "hard light" method I use a "soft light" method. the matplotlib is converting the RGB colors to HSV and then calculate the new saturation and value according to the intensity. I use a formula based on the description of ImageMagick's pegtop_light.which is much faster as it is a single formula. Another advantage is the option to use a separate layer as the intensity and another as the data used for colors.
The modified functions are hillshade and set_shade as follows:
Example of use:
One can save the code to a file named say: shading.py
now say we have a 4 byte float DEM data in a 560 lines 420 samples binary file. in a python code:
from pylab import *
from shading import set_shade
from shading import hillshade
dem = fromfile('DEM.dem',dtype=float32).reshape(560,420)
rgb = set_shade(dem,cmap=cm.gist_earth)
imshow(rgb)
will produce the "my shade method" image as above.
say we have a data to be plot using the DEM data as intensity:
replace the line before last with:
rgb = set_shade(data,intensity=hillshade(dem),cmap=cm.gist_earth)
The problem is it uses the data itself as intensity and data. It is very useful for viewing a DEM but sometimes you would like the DEM as intensity underlying some other data. Another problem is that the shade method is producing a very light colored image sometimes even white where intensity is high.
I used as an example a DEM derived from SRTM v4 data acquired at the International Centre for Tropical Agriculture (CIAT - http://srtm.csi.cgiar.org) the hillshade production was made using LightSource class with azimuth of 165 deg. and altitude of 45 deg.)
 |  | |
| DEM - gist_earth color scheme | Hill-shade (azdeg-165,altdeg-45) | |
 |  | |
| matplotlib shade method | My shade method |
The modified functions are hillshade and set_shade as follows:
#!/bin/env python
from pylab import *
def set_shade(a,intensity=None,cmap=cm.jet,scale=10.0,azdeg=165.0,altdeg=45.0):
''' sets shading for data array based on intensity layer
or the data's value itself.
inputs:
a - a 2-d array or masked array
intensity - a 2-d array of same size as a (no chack on that)
representing the intensity layer. if none is given
the data itself is used after getting the hillshade values
see hillshade for more details.
cmap - a colormap (e.g matplotlib.colors.LinearSegmentedColormap
instance)
scale,azdeg,altdeg - parameters for hilshade function see there for
more details
output:
rgb - an rgb set of the Pegtop soft light composition of the data and
intensity can be used as input for imshow()
based on ImageMagick's Pegtop_light:
http://www.imagemagick.org/Usage/compose/#pegtoplight'''
if intensity is None:
# hilshading the data
intensity = hillshade(a,scale=10.0,azdeg=165.0,altdeg=45.0)
else:
# or normalize the intensity
intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
# get rgb of normalized data based on cmap
rgb = cmap((a-a.min())/float(a.max()-a.min()))[:,:,:3]
# form an rgb eqvivalent of intensity
d = intensity.repeat(3).reshape(rgb.shape)
# simulate illumination based on pegtop algorithm.
rgb = 2*d*rgb+(rgb**2)*(1-2*d)
return rgb
def hillshade(data,scale=10.0,azdeg=165.0,altdeg=45.0):
''' convert data to hillshade based on matplotlib.colors.LightSource class.
input:
data - a 2-d array of data
scale - scaling value of the data. higher number = lower gradient
azdeg - where the light comes from: 0 south ; 90 east ; 180 north ;
270 west
altdeg - where the light comes from: 0 horison ; 90 zenith
output: a 2-d array of normalized hilshade
'''
# convert alt, az to radians
az = azdeg*pi/180.0
alt = altdeg*pi/180.0
# gradient in x and y directions
dx, dy = gradient(data/float(scale))
slope = 0.5*pi - arctan(hypot(dx, dy))
aspect = arctan2(dx, dy)
intensity = sin(alt)*sin(slope) + cos(alt)*cos(slope)*cos(-az - aspect - 0.5*pi)
intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
return intensity
from pylab import *
def set_shade(a,intensity=None,cmap=cm.jet,scale=10.0,azdeg=165.0,altdeg=45.0):
''' sets shading for data array based on intensity layer
or the data's value itself.
inputs:
a - a 2-d array or masked array
intensity - a 2-d array of same size as a (no chack on that)
representing the intensity layer. if none is given
the data itself is used after getting the hillshade values
see hillshade for more details.
cmap - a colormap (e.g matplotlib.colors.LinearSegmentedColormap
instance)
scale,azdeg,altdeg - parameters for hilshade function see there for
more details
output:
rgb - an rgb set of the Pegtop soft light composition of the data and
intensity can be used as input for imshow()
based on ImageMagick's Pegtop_light:
http://www.imagemagick.org/Usage/compose/#pegtoplight'''
if intensity is None:
# hilshading the data
intensity = hillshade(a,scale=10.0,azdeg=165.0,altdeg=45.0)
else:
# or normalize the intensity
intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
# get rgb of normalized data based on cmap
rgb = cmap((a-a.min())/float(a.max()-a.min()))[:,:,:3]
# form an rgb eqvivalent of intensity
d = intensity.repeat(3).reshape(rgb.shape)
# simulate illumination based on pegtop algorithm.
rgb = 2*d*rgb+(rgb**2)*(1-2*d)
return rgb
def hillshade(data,scale=10.0,azdeg=165.0,altdeg=45.0):
''' convert data to hillshade based on matplotlib.colors.LightSource class.
input:
data - a 2-d array of data
scale - scaling value of the data. higher number = lower gradient
azdeg - where the light comes from: 0 south ; 90 east ; 180 north ;
270 west
altdeg - where the light comes from: 0 horison ; 90 zenith
output: a 2-d array of normalized hilshade
'''
# convert alt, az to radians
az = azdeg*pi/180.0
alt = altdeg*pi/180.0
# gradient in x and y directions
dx, dy = gradient(data/float(scale))
slope = 0.5*pi - arctan(hypot(dx, dy))
aspect = arctan2(dx, dy)
intensity = sin(alt)*sin(slope) + cos(alt)*cos(slope)*cos(-az - aspect - 0.5*pi)
intensity = (intensity - intensity.min())/(intensity.max() - intensity.min())
return intensity
Example of use:
One can save the code to a file named say: shading.py
now say we have a 4 byte float DEM data in a 560 lines 420 samples binary file. in a python code:
from pylab import *
from shading import set_shade
from shading import hillshade
dem = fromfile('DEM.dem',dtype=float32).reshape(560,420)
rgb = set_shade(dem,cmap=cm.gist_earth)
imshow(rgb)
will produce the "my shade method" image as above.
say we have a data to be plot using the DEM data as intensity:
replace the line before last with:
rgb = set_shade(data,intensity=hillshade(dem),cmap=cm.gist_earth)
Thursday, December 10, 2009
create a grd file from DEM
After creating a DEM file using SRTM.py script, now say we need to convert it to a GMT .grd file.
1. convert the binary file to a grd file:
xyz2grd [binary file] -G[output_grd_file] -Zh -I[pixel_size] -R[west]/[south]/[east]/[north]r
- don't forget the r at the end.
- in the -Zh, h is for 2 byte integer. change if needed
2. Sample to a different pixel size and sub-region:
grdsample [input_grd_file] -G[output_grd_file] -I[new_pixel_size] -R[west]/[south]/[east]/[north]r
- if a known number of pixels is desired, say in east-west, use: east = west+[needed_pixel_number - 1]*[new_pixel_size]
3. put zeros instead of null values -9999:
grdmath [input_grd_file] -9999 GT [input_grd_file] MUL = [output_grd_file]
- note that if you sample some of the null pixels will be with different values so use say -500 instead of -9999 as a threshold.
1. convert the binary file to a grd file:
xyz2grd [binary file] -G[output_grd_file] -Zh -I[pixel_size] -R[west]/[south]/[east]/[north]r
- don't forget the r at the end.
- in the -Zh, h is for 2 byte integer. change if needed
2. Sample to a different pixel size and sub-region:
grdsample [input_grd_file] -G[output_grd_file] -I[new_pixel_size] -R[west]/[south]/[east]/[north]r
- if a known number of pixels is desired, say in east-west, use: east = west+[needed_pixel_number - 1]*[new_pixel_size]
3. put zeros instead of null values -9999:
grdmath [input_grd_file] -9999 GT [input_grd_file] MUL = [output_grd_file]
- note that if you sample some of the null pixels will be with different values so use say -500 instead of -9999 as a threshold.
Friday, October 2, 2009
Reading GMT .GRD/NetCDF file using python
When reading GTM file in GRD format you actually read a NetCDF format.
using python and pylab to read it enable matplotlib to plot it.
say we have a grd file named z.grd which contains 3 variables: x,y and z.
reading z values in python is done by the following three lines:
from pylab import *
from scipy.io import netcdf_file as netcdf
data = netcdf('z.grd','r').variables['z'][::-1]
Now data is a numpy array containing z values in the file dimensions.
for more info look at: http://gfesuite.noaa.gov/developer/netCDFPythonInterface.html
using python and pylab to read it enable matplotlib to plot it.
say we have a grd file named z.grd which contains 3 variables: x,y and z.
reading z values in python is done by the following three lines:
from pylab import *
from scipy.io import netcdf_file as netcdf
data = netcdf('z.grd','r').variables['z'][::-1]
Now data is a numpy array containing z values in the file dimensions.
for more info look at: http://gfesuite.noaa.gov/developer/netCDFPythonInterface.html
Tuesday, September 29, 2009
howto edit multiple files via command line
In order to replace a regexp in multiple files using the command line use:
sed -i -e 's/regexp/new_regexp/' file_name
where file_name can be also a regexp (e.g. *.txt)
sed -i -e 's/regexp/new_regexp/' file_name
where file_name can be also a regexp (e.g. *.txt)
Wednesday, July 22, 2009
Hard Drive Resurrection
July 22, 2009: One of my hard drives turned out to be dead.
When I disconnected it from the computer - all is well, the computer is working fine without the missing HD, only I had very important data on it and must resurrect it.
when I connect it - the whole system is freezing So I belive it is something to do with the logic board. I googled it up and came upon this.
Basically, I need another HD with the same properties (manufacturer, model,firmware) and just switch the logic board which should be easy and hopefully work.
I now need to find a similar HD for the replacement.
July 23, 2009: Well, no luck in finding a similar HD and sending the HD to data recovery is more than 150$. Before I spend so much money I would like to check the cheapest option:
Freezing the HD. lets see what happens...
July 23, 2009, supplemental: it worked!
I was able to recover the data after few hours in the freezer. I hardly believed it, but it actually worked. I'll now spend my money buying a larger HD.
wooohooo!
When I disconnected it from the computer - all is well, the computer is working fine without the missing HD, only I had very important data on it and must resurrect it.
when I connect it - the whole system is freezing So I belive it is something to do with the logic board. I googled it up and came upon this.
Basically, I need another HD with the same properties (manufacturer, model,firmware) and just switch the logic board which should be easy and hopefully work.
I now need to find a similar HD for the replacement.
July 23, 2009: Well, no luck in finding a similar HD and sending the HD to data recovery is more than 150$. Before I spend so much money I would like to check the cheapest option:
Freezing the HD. lets see what happens...
July 23, 2009, supplemental: it worked!
I was able to recover the data after few hours in the freezer. I hardly believed it, but it actually worked. I'll now spend my money buying a larger HD.
wooohooo!
Monday, May 25, 2009
HowTo view a ROI PAC file? (or "replacement for mdx")
The Problem:
I can't get mdx/dgx in order to view my ROI PAC files.
The Solution:
a Python script called roiview (Get RoiView at) wich is one of my other scripts.
This script uses pylab and can read, display and save images of ROI PAC quite similar to mdx/dgx.
it can handle *.unw, *.cor, *.slc, *.int and *.dem files.
just download, extract and use:
"roiview -h" for manual.
Update! Oct 13, 2010: A new version of RoiView (0.75) is now available at:
I can't get mdx/dgx in order to view my ROI PAC files.
The Solution:
a Python script called roiview (Get RoiView at
) wich is one of my other scripts.This script uses pylab and can read, display and save images of ROI PAC quite similar to mdx/dgx.
it can handle *.unw, *.cor, *.slc, *.int and *.dem files.
just download, extract and use:
"roiview -h" for manual.
Update! Oct 13, 2010: A new version of RoiView (0.75) is now available at:
Wednesday, May 13, 2009
HowTo Create a SRTM DEM for ROI PAC Processing
When Processing interferograms, usually, you will also require a DEM. The NASA's SRTM project is freely available from different places in different formats and levels. The latest version for now (IV) is available from CGIAR Consortium for Spatial Information.
in order to ease the DEM production for a research, I wrote a small Python script. The script will create a DEM with 1 deg to each side of the center of scene requested. It will create a ROI PAC header as well. there is an option to create a JPG image of shaded relief of the imag. You will need Python to be installed and also PyLab and PIL modules.
simply download SRTM.py (zip,2.9kb) from my scripts page, change permission to execute, and run it...
Don't forget to credit CGIAR for the data. Citations should be made as follows: Jarvis A., H.I. Reuter, A. Nelson, E. Guevara, 2008, Hole-filled seamless SRTM data V4, International Centre for Tropical Agriculture (CIAT), available from http://srtm.csi.cgiar.org/
in order to ease the DEM production for a research, I wrote a small Python script. The script will create a DEM with 1 deg to each side of the center of scene requested. It will create a ROI PAC header as well. there is an option to create a JPG image of shaded relief of the imag. You will need Python to be installed and also PyLab and PIL modules.
simply download SRTM.py (zip,2.9kb) from my scripts page, change permission to execute, and run it...
Don't forget to credit CGIAR for the data. Citations should be made as follows: Jarvis A., H.I. Reuter, A. Nelson, E. Guevara, 2008, Hole-filled seamless SRTM data V4, International Centre for Tropical Agriculture (CIAT), available from http://srtm.csi.cgiar.org/
Thursday, May 7, 2009
Using HighLearn & Linux
The Problem:
Can't use HighLearn on Linux with firefox.
This e-learning system of some of the academic institutions in Israel is Microsoft explorer based. It is a very unfortunate setting for linux users. I hope they will change the system soon. Probably they won't...
The Solution:
Thanks to a tip from a friend, I google it up and discovered this thread (Hebrew) which suggest using konqueror.
Just install Greasemonkey on your firefox, restart firefox and click "Install" Here.
Note! not all the features are working yet.
Can't use HighLearn on Linux with firefox.
This e-learning system of some of the academic institutions in Israel is Microsoft explorer based. It is a very unfortunate setting for linux users. I hope they will change the system soon. Probably they won't...
The Solution:
Thanks to a tip from a friend, I google it up and discovered this thread (Hebrew) which suggest using konqueror.
Just install Greasemonkey on your firefox, restart firefox and click "Install" Here.
Note! not all the features are working yet.
Monday, May 4, 2009
Howto view an image in Google Earth
The Problem:
I have an image I wish to view on Google Earth.
The Solution:
I assume you have an image in jpg or tiff or gif format. and you have a world file for it or can create one. Then, all you need to do is to run the image2kml.csh script.
just download (~1kb), apply execution privilege (chmod +x [ScriptFileName])
and run it. or visit my scripts page
Tip:
gif format image can support transparency you can use GIMP or other image manipulation program to turn unwanted parts of the image to be transparent so you can see the Google Earth layover under the image.
I have an image I wish to view on Google Earth.
The Solution:
I assume you have an image in jpg or tiff or gif format. and you have a world file for it or can create one. Then, all you need to do is to run the image2kml.csh script.
just download (~1kb), apply execution privilege (chmod +x [ScriptFileName])
and run it. or visit my scripts page
Tip:
gif format image can support transparency you can use GIMP or other image manipulation program to turn unwanted parts of the image to be transparent so you can see the Google Earth layover under the image.
Howto share folders from windows to Linux
The Problem:
I use two machines, one is a Linux based and one windows based. I need to share files between them.
The Solution:
One option is to use a network hard drive. It might be quite slow though.
Other option is to use windows share documents option. It is not the safest methods, but surely one of the easiest.
On Windows:
- Right click on the folder to be shared and select “sharing and Security...”
- mark “Share this folder on the network”
- give a network name – best if only one word (it is not case sensitive)
- if you will need to change the files and not only to view them mark also “Allow network users to change my files”
On Linux:
Gui solution:
- go to “Places” -> “Connect
to Server..."
- switch to “Widows Share” on service type
- On “Server:” enter the windows machine IP number
- On “Share:” enter the name of the shared folder as was entered on the Windows sharing options
- if you use a different user name enter your windows user name on “User Name:”
- You can use an alias for this folder on the Linux machine by entering a new name at “Bookmark name:”
Now you can see and use the folder and files.
Or using mount (not Gui):
- create a new mount point (e.g. /mnt/windowsshare)
- Use mount command:
mount -t cifs //serverIP/shareName -o username=WindowsUserName,password=WindowsPassword /mnt/windowsshare
Or edit /etc/fstab file:
mounting windows share:
//XXX.XX.XX.XX/shareName /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]the file should contain the remote system properties:username=[username]password=[userpassword]
Tuesday, April 28, 2009
Whoto know if a long process is done?
The Problem:
I sometimes run a very long process which can even reach up to few days.
Instead of looking and checking if the process has done, I needed a solution to get a message from the process itself.
One way to do it is by using mail like this:
echo "some message" | mail -s "subject of mail" yourmail@wherever
Unfortunately, for some reason it doesn't work for me.
if you have any solution - let me know. I used a workaround...
The Solution (Option I):
Using Thunderbird and cshell and a tip from here (look at the comments), I built a script that can be added to a command line or another script which run's the process.
needed ingredients:
- Thunderbird
- xdotools
usage: simply copy the command below, edit where needed and run:
mail_script.csh [few message body words]
mail_script.csh:
Option I needs Thunderbird to be installed and it uses gui so somtimes there are conflicts with user activity. an alternative solution is to install ssmtp (most Linux distros has it as a package). after installing edit /etc/ssmtp/ssmtp.conf file:
most importent lines to edit:
Mailhub - you smtp server (e.g smtp.gmail.com or any other smtp server)
AuthUser - see conf file link above
AuthPass - see conf file link above
AuthMethod - see conf file link above
FromLineOverride - remove the # at the start of line
edit /etc/ssmtp/revaliases:
add a line:
root:[user]@[mailaccount]:[smtpserver]:[port]
and run the script mail_script.csh:
# this will send a mail with subject:"mission accomplished"
# and command line varibles (argv) as the message body
# the script requier ssmtp installed
# Just change the -F , SendTo and "To" fields in the command below and set a program to run it at the end of process
#
# by Ran Novitsky Nof 2010 @ BGU
set body = `echo $argv`
sudo ssmtp -F"name of sender" recipient@mailserver << END
To:recipient@mailserver
Subject:Mission accomplished
$body
END
I sometimes run a very long process which can even reach up to few days.
Instead of looking and checking if the process has done, I needed a solution to get a message from the process itself.
One way to do it is by using mail like this:
echo "some message" | mail -s "subject of mail" yourmail@wherever
Unfortunately, for some reason it doesn't work for me.
if you have any solution - let me know. I used a workaround...
The Solution (Option I):
Using Thunderbird and cshell and a tip from here (look at the comments), I built a script that can be added to a command line or another script which run's the process.
needed ingredients:
- Thunderbird
- xdotools
usage: simply copy the command below, edit where needed and run:
mail_script.csh [few message body words]
mail_script.csh:
#!/bin/cshThe Solution (Option II):
# this will send a mail with subject:"mission accomplished"
# and command line variables (argv) as the message body
# the script require Thunderbird and xdotool installed
# Just change the "to" field in the command below and
# set a program to run it at the end of process
# by Ran Novitsky Nof 2009 @ BGU
set body = `echo $argv`
thunderbird -compose "to='yourmail@wherever',subject='Mission Accomplished',body=$body" & ; sleep 5 ; set WID=`xdotool search --title
"Compose" head -1`; xdotool windowfocus $WID;xdotool key ctrl+Return
Option I needs Thunderbird to be installed and it uses gui so somtimes there are conflicts with user activity. an alternative solution is to install ssmtp (most Linux distros has it as a package). after installing edit /etc/ssmtp/ssmtp.conf file:
most importent lines to edit:
Mailhub - you smtp server (e.g smtp.gmail.com or any other smtp server)
AuthUser - see conf file link above
AuthPass - see conf file link above
AuthMethod - see conf file link above
FromLineOverride - remove the # at the start of line
edit /etc/ssmtp/revaliases:
add a line:
root:[user]@[mailaccount]:[smtpserver]:[port]
and run the script mail_script.csh:
# this will send a mail with subject:"mission accomplished"
# and command line varibles (argv) as the message body
# the script requier ssmtp installed
# Just change the -F , SendTo and "To" fields in the command below and set a program to run it at the end of process
#
# by Ran Novitsky Nof 2010 @ BGU
set body = `echo $argv`
sudo ssmtp -F"name of sender" recipient@mailserver << END
To:recipient@mailserver
Subject:Mission accomplished
$body
END
Tag Clouds
The Problem:
How to replace the tag list (aka labels) to a tag cloud?
The Solution:
Details in here you can see the outcome on the sidebar of my blog.
Tip:
To add a heading to the tags cloud simply add another <div> before the div of widget-content:
new div: <div><h2 class="'title'">Labels Cloud</h2></div> (in red is the title I selected for the gadget, can be anything actually)
original script (only the first few lines):<b:widget id="'Label1'" locked="'false'" title="'Labels'" type="'Label'"><b:includable id="'main'"> [new div in here] <div class="'widget-content'"><div id="'LabelDisplay'/"></div><script language="'javascript'" type="'text/javascript'">
How to replace the tag list (aka labels) to a tag cloud?
The Solution:
Details in here you can see the outcome on the sidebar of my blog.
Tip:
To add a heading to the tags cloud simply add another <div> before the div of widget-content:
new div: <div><h2 class="'title'">Labels Cloud</h2></div> (in red is the title I selected for the gadget, can be anything actually)
original script (only the first few lines):<b:widget id="'Label1'" locked="'false'" title="'Labels'" type="'Label'"><b:includable id="'main'"> [new div in here] <div class="'widget-content'"><div id="'LabelDisplay'/"></div><script language="'javascript'" type="'text/javascript'">
Monday, April 27, 2009
Orbits and other auxiliary data
I use Delft orbital data (ODR) which is available from here. don't forget to use the precise version and to download all .ODR files (make sure it is not .odr after downloading) and the arclist. also, download and install getorb software and link or move the executables to the ROI_PAC binaries directory.
I also use DORIS orbital data for Envisat which is available from here. but will require a user and password. the DORIS data is used when no ODR data is yet to be available for new Envisat images.
for Envisat I also use auxiliary data available from here.
in my lab, all the orbital data is in ~/ROI_PAC_3_0/Delft with directory tree as follows:
in ~/ROI_PAC_3_0/Delft I put the Envisat auxiliary data (INS and XCA files)
I also use DORIS orbital data for Envisat which is available from here. but will require a user and password. the DORIS data is used when no ODR data is yet to be available for new Envisat images.
for Envisat I also use auxiliary data available from here.
in my lab, all the orbital data is in ~/ROI_PAC_3_0/Delft with directory tree as follows:
in ~/ROI_PAC_3_0/Delft I put the Envisat auxiliary data (INS and XCA files)
- doris/Envisat/YYYYMM contains orbital data for the year (YYYY) and month (MM)
- Envisat contains Envisat Delft orbit data (.ODR .res arclist files)
- ERS1 contains ERS1 Delft orbit data (.ODR .res arclist files)
- ERS2 contains ERS2 Delft orbit data (.ODR .res arclist files)
the files must be updated from time to time as more images are added to the list.
ROI PAC installation
UPDATE: Oct 9, 2011 (for version 3.0.1):
first download ROI-PAC and test data from open channel Foundation
I will assume the tar files are extracted at home directory (~), but it can be anywhere you wish.
don't forget to grand read/write access to relevant users.
make sure you have at least gcc and gfortran installed on your system.
go to ~/ROI_PAC_3_0_1/ROI_PAC/ directory.
follow steps described in ~/ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC up until line:
which uncompress
make sure you have a TEST DIR testing data to test the compiled program. this directory should be tared but not gzipped using:
gunzip roi_pac_test_dir.tar.gz
this will convert the roi_pac_test_dir.tar.gz to roi_pac_test_dir.tar
then create a directory test-runs in the ROI_PAC directory.
cd test_runs
and run the multitest.sh script:
../contrib/multitest.sh /FULLPATH/roi_pac_testdir.tar /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/share/roi_pac /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILD1/bin ... ... ... /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILDN/bin
where:
FULLPATH is the path to your home directory (or where ever the install is)
xxxxxx-yyyy is the multibuild date-time stamp
and BUILD1 to BUILDN are all the compilers sub directories in /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/ e.g. defaults, gfortran, gfortran64 etc.
check for the fastest build by checking the batchlog files for each build in test-runs directory.
go to multibuild directory and into installs. get the bin directory from the desired compiler directory and copy it to the ~/ROI_PAC_3_0_1/ROI_PAC directory.
look also at ROI_PAC Wiki for updates and patches.
now you can use ROI PAC, but keep in mined orbits and other auxiliary data should also be available for processing.
first download ROI-PAC and test data from open channel Foundation
I will assume the tar files are extracted at home directory (~), but it can be anywhere you wish.
don't forget to grand read/write access to relevant users.
make sure you have at least gcc and gfortran installed on your system.
go to ~/ROI_PAC_3_0_1/ROI_PAC/ directory.
follow steps described in ~/ROI_PAC_3_0_1/ROI_PAC/AAREADME_BUILD_ROIPAC up until line:
which uncompress
make sure you have a TEST DIR testing data to test the compiled program. this directory should be tared but not gzipped using:
gunzip roi_pac_test_dir.tar.gz
this will convert the roi_pac_test_dir.tar.gz to roi_pac_test_dir.tar
then create a directory test-runs in the ROI_PAC directory.
cd test_runs
and run the multitest.sh script:
../contrib/multitest.sh /FULLPATH/roi_pac_testdir.tar /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/share/roi_pac /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILD1/bin ... ... ... /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/BUILDN/bin
where:
FULLPATH is the path to your home directory (or where ever the install is)
xxxxxx-yyyy is the multibuild date-time stamp
and BUILD1 to BUILDN are all the compilers sub directories in /FULLPATH/ROI_PAC_3_0_1/ROI_PAC/multibuild-xxxxxx-yyyy/installs/ e.g. defaults, gfortran, gfortran64 etc.
check for the fastest build by checking the batchlog files for each build in test-runs directory.
go to multibuild directory and into installs. get the bin directory from the desired compiler directory and copy it to the ~/ROI_PAC_3_0_1/ROI_PAC directory.
- note: I moved the bin directory to: ~/ROI_PAC_3_0/ROI_PAC/BIN/LIN
look also at ROI_PAC Wiki for updates and patches.
now you can use ROI PAC, but keep in mined orbits and other auxiliary data should also be available for processing.
Installing Linux Fedora Core 10
Most of the info can be found here
I will only add some relevant steps to InSAR processing as I use it and steps I found useful.
installing gnome and most of the distro (including tcsh as it is my favorite shell)
- change nautilus to open always in browser
- remove SElinux policy
- remove option in keyboard options - allow mouse to be controlled by keypad
- add vino vnc option to user
- change autologin for user: vi /etc/gdm/gdm.schemas
- add user to sudo:
su --login -c 'visudo'
Now go below the line:
root ALL=(ALL) ALL
press a, and type the following
username ALL=(ALL) ALL
where username is the username you use to login. Next press Escape.
Now, if you want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) ALL
and with your cursor pointing on # press x
If you don't want to be prompted for your root password each time you use the sudo command go to this line:
# %wheel ALL=(ALL) NOPASSWD: ALL
and with your cursor pointing on # press x
Next, press :wq to save and exit. (type the : as well)
* Adding your user to the wheel group
Just type:
su -c 'gpasswd -a username wheel'
* Testing sudo
To test if you have done it correctly as a simple user type:
$ sudo whoami
If everything is working properly the command should return the word 'root'.
- adding windows fonts:
wget http://www.my-guides.net/en/images/stories/fedora10/msttcore-fonts-2.0-2.noarch.rpm
su -c 'rpm -ivh msttcore-fonts-2.0-2.noarch.rpm'
- open terminal here in nautilus
su -c 'yum install nautilus-open-terminal'
- Enable Num Lock on GNOME startup
su -c 'yum install numlockx'
su -c 'cp /etc/gdm/Init/Default /etc/gdm/Init/Default_backup'
su -c 'gedit /etc/gdm/Init/Default'
And add the following lines at the end of the file before the exit 0.
if [ -x /usr/bin/numlockx ]; then
/usr/bin/numlockx on
fi
- add to install:
yum extender
yum Fastest Mirror Plugin
GMT
qgis
grace
filezilla
ipython
KDocker
google-earth
wine
g77
bluecurve icons
gnash - for flash viewing
- setting /etc/X11/xorg.conf for display with no monitor attached:
make sure to use a video card which is not onboard (or perhaps this is only valid for the 965 intel mobo?)
- edit /etc/fstab for mounting external HD (LAN):
//XXX.XX.XX.XX/DATA /mountpoint cifs auto,owner,rw,username=XXX,uid=XXX,gid=XXX 0 0
NOTE! - it will not mount on startup. need to do: "sudo mount mountpoint" to mount. one can add password=[user-passwd] after username=[username] or instead username and password : credentials=[path to a file containing user and password]
the file should contain the remote system properties:
username=[username]
password=[userpassword]
- adding python modules
for gui: http://easygui.sourceforge.net/index.html
scipy
numpy
matplotlib
ipython
pylab
in a case of emergency: enabling write on a system error:
mount -t ext3 /dev/sda1 / -o remount,rw
- installing ROI PAC
- adding axillary data
Working on two machines at once
The Problem:
On my work I use two computers, one is Linux based which carry out all the heavy processing and another is windows based for viewing results, office related jobs and other Microsoft compatibility issues.
My problem was I had two mice, two keyboard and two screens. it is very confusing and tiring to switch from one to another.
The Solution:
The Linux distro comes with a wonderful component - VINO SERVER a VNC server which connects to your display 0 (the main display).
so I installed REALVNC viewer on my windows computer, connected one screen to the DVI exit and one to the D-sub (VGA) exit on the graphic card.
now, all need to do is to enter the Linux machine via VNC, drag the window to the second screen and use "Full Screen" view. This allows me to use one set of keyboard and mouse and move from one machine to another just by moving from one screen to another.
Tips:
On my work I use two computers, one is Linux based which carry out all the heavy processing and another is windows based for viewing results, office related jobs and other Microsoft compatibility issues.
My problem was I had two mice, two keyboard and two screens. it is very confusing and tiring to switch from one to another.
The Solution:
The Linux distro comes with a wonderful component - VINO SERVER a VNC server which connects to your display 0 (the main display).
so I installed REALVNC viewer on my windows computer, connected one screen to the DVI exit and one to the D-sub (VGA) exit on the graphic card.
now, all need to do is to enter the Linux machine via VNC, drag the window to the second screen and use "Full Screen" view. This allows me to use one set of keyboard and mouse and move from one machine to another just by moving from one screen to another.
Tips:
- autologin:
Since you can't "see" the Linux machine without connecting to it, you can tweak the bootloader to load you user automatically:
edit /etc/gdm/custom.conf (as superuser if needed) and change:
[daemon]
TimedLoginEnable=true
TimedLogin=yourusername
TimedLoginDelay=0 - ssh first:
if you don't want to use autologin for security or other reason, you can login via ssh and start a gnome session and then use the VNC viewer. just enable ssh daemon and vino server. - othere VNC servers:
you can always use different VNC server and connect to other displays then 0. - graphic card:
I tried to use this method with an onboard G. card and only got low resolution screen size.
it might be my mother board (intel DQ965GF) or bad X settings but using a GeForce card I managed to use my desired resolution with no problems.
Welcom to My Blog.
My name is Ran Novitsky Nof. I currently work at the Geophysical Institute of Israel.
This blog's first aim is to serve as a holding place for my ideas, thoughts, tips and workflows during my research and sharing it with the world - if only one person would benefit from something here - I did my share.
Additional Info may be found at my personal homepages : ran.rnof.info
This blog's first aim is to serve as a holding place for my ideas, thoughts, tips and workflows during my research and sharing it with the world - if only one person would benefit from something here - I did my share.
Additional Info may be found at my personal homepages : ran.rnof.info
Subscribe to:
Posts (Atom)