Scraping website for Notifications using Python (Attempt made on VITEEE 2014 slot booking page)

Scraping websites where in the actual information is not published before a stipulated time, is necessary. For example a recent attempt to book time slots for an examination that my sister was yet to take brought forward some insights and needs. Tasks such as Tatkal ticket booking, exam slot booking, trying to open the Results page of some exam ...etc are all time critical jobs . Since these are all First-Come-First-Serve based tasks, being the first one to open the website is crucial. I was met with a similar challenge today wherein the time slot booking for VITEEE 2014 was to start today at 10:30PM. However, the website link pointed to the same webpage with some instructions even after 11:00PM. As I said, these are time critical applications and this particular one wasn't a case of the webpage not opening but more of a case of the webpage getting updated. So pinging the website won't serve the purpose. So a quick 5min script in python using the BeautifulSoup module solved the problem. The code is simple and obvious for most python hobbyists but the exposed possibilities are many. The Beep API via pywin32 was used to alert me when the webpage gets updated by playing a beep. The cue that hinted the update of the webpage was a change in the text part of the page. As it was coded in a very short time not a much attention was paid to the exceptions that might arise. Just wanted to get the work done. Only thing to keep in mind is that one must be reasonable in setting the ping time. Setting it too low might overload the server which I need not tell you. This isn't something big but was something that works and also definitely worth sharing.

 

Part 2: Fast Object Tracking in Python using OpenCV

In my previous post I've blabbered about image processing in python using OpenCV libraries. The setup process was discussed and I also went into some basics of reading a image into a variable and showing that image in a window. Now to get into the actual stuff.


The below code is the one from the previous post.
import cv
img = cv.LoadImage("C:\Users\Public\Pictures\Sample Pictures\Koala.jpg")
cv.NamedWindow("hi",0)
while True:
    cv.ShowImage("hi",img)
    k=cv.WaitKey(100)
    if k==27:
        break
cv.DestroyAllWindows()

It loads an image from one of your folders and displays it in a new window. And a correction, you needn't use '\\' instead of '\'. The latter will work fine. We plan on tracking an object based on its color. So we need to isolate it. The typical image data is a combination of three 2D arrays.

The different Layers of an image may be seen above

Now that is how the image data is arranged in rows and columns.As you can see above, there are three layers for each color of the primary color set.  The numerical values in the rows and columns of the array give the shade of Blue,Green or Red to be used. 

Note: The layers are in the order of BGR in OpenCV unlike in Matlab where they are in the order RGB.If it doesn't mean anything to you, forget I said it. 

The HSI color model is just another way of visualizing image data. Below is given the HSI color space

However, this is not the only way the image data may be arranged. It may also be arranged using a HSI color space. Allow me to explain.

As you can see here, the Hue(H) gives the color in specific, the Saturation(S) gives how thick the color has to be and Intensity(I) gives how different it is from black. We are not interested in the S or I, all that we are worried about is the color, so the Hue(H).

When an image is read ,it is read in the RGB format. To perform color based object tracking we need to change t he color space from RGB to HSI

Hue Bar

For that purpose we use a OpenCV function cv.CvtColor(src,dst,conversion_type) to convert the color format to HSI.

The CvtColor function is capable of doing a lot of color conversions. The third parameter defines the type of conversion. 

Before conversion we need to create an another image variable to hold the new HSI image and for that we use cv.CreateImage() function. It takes three parameters-Size of the image,bit depth, channels(one each for H,S,I). The cv.GetSize() function may be used to get the size of an image. The size of the destination image must be same as that of src image.

>>>img_hsi=cv.CreateImage(cv.GetSize(img),8,3)

The above statement simply create a 2D image variable with three layers(one each for H,S,I).

Now for the actual conversion we use CvtColor function as below

>>>cv.CvtColor(img,img_hsi,cv.CV_RGB2HSV) # Intensity(I) is sometimes called Value(V)

the resulting HSI version of 'img' is now copied into 'img_hsi' variable.We may view it using ShowImage function as below. The end code is given below:

import cv
img = cv.LoadImage("C:\Users\Public\Pictures\Sample Pictures\Koala.jpg")
cv.NamedWindow("hi",0)


#lines in addition to previous code
img_hsi=cv.CreateImage(cv.GetSize(img),8,3)
cv.CvtColor(img,img_hsi,cv.CV_RGB2HSV)


while True:
    cv.ShowImage("hi",img_hsi)
    k=cv.WaitKey(100)
    if k==27:
        break
cv.DestroyAllWindows()

The Image appears the same but now it is being read as an HSI image instead like an RGB. To be sure of that let's see all the laye and srs separately. To split the image data by layers we use cv.Split(src,dst0,dst1,dst2,dst3).  So, the layers H,S,I of the image src are split and saved into dst0,ds1,ds2,dst3 in the same order.


Before that we create three images with one layer (dst0,dst1,dst2).Let's do it.
>>>img_h=cv.CreateImage(cv.GetSize(img),8,1) #'1' means it has only one layer
                                                                         # Similarly for others
>>>img_s=cv.CreateImage(cv.GetSize(img),8,1)

>>>img_i=cv.CreateImage(cv.GetSize(img),8,1)


Now to split and store the individual layers and copy them into the respective variables.

>>>cv.Split(img_hsi,img_h,img_s,img_i,None) # it takes in 4 arguments so the fourth       # argument is None as we have just 3 layers 

Viewing the invidual layers in separate windows.
>>>cv.ShowImage("Hue Layer",img_h)
>>>cv.ShowImage("Saturation Layer",img_s)
>>>cv.ShowImage("Intensity Layer",img_i)


Tip: You need not create windows every time to display an image. Instead just use different names for windows in the ShowImage("window_name",img_var) function. The windows are created automatically. Just don't forget to use the DestroyAllWindows() function to destroy all the created windows at the end.

The complete code is given below: 

import cv

img=cv.LoadImage("C:\Users\Public\Pictures\Sample Pictures\Koala.jpg")
img_hsi=cv.CreateImage(cv.GetSize(img),8,3)
cv.CvtColor(img,img_hsi,cv.CV_RGB2HSV);


img_h=cv.CreateImage(cv.GetSize(img),8,1)
img_s=cv.CreateImage(cv.GetSize(img),8,1)
img_i=cv.CreateImage(cv.GetSize(img),8,1)
cv.Split(img_hsi,img_h,img_s,img_i,None)


while True:   
    cv.ShowImage("hue",img_h)
    cv.ShowImage("sat",img_s)
    cv.ShowImage("val",img_i)
    k=cv.WaitKey(100)
    if k==27:break;
    
cv.DestroyAllWindows()

Intensity Layer

The intensity Layer pixel values give the brightness values at various pixel locations. The HSI model is often used to isolate separate this layer which accounts for the lighting conditions.As a thumb rule, we must always try to remove the effect of lighting conditions on the image.

  

Saturation Layer

                    

Hue Layer

The Hue Layer gives the colors at various picture locations. Look at the HSI cone above.

The same code may be implemented for real time images using the second code snippet as follows:

#Under Construction.
The above code is self explanatory,provided you went through the previous post. Thats it for now

Fast Object Tracking in Python using OpenCV

Machine Vision based competitions are being held in and around various engineering colleges. So, I have decided to make a small tutorial on how to make a small image processing application in python.


It tracks an object based on its color. A lot of people think of Matlab when they hear anything remotely related to image processing. However,there are a lot more ways of performing image processing on a PC. 

Some knowledge of python is appreciated before going into the details. It doesn't take a lot of time and you may learn the basics from my previous post (one of the most well spent moments of your life)


For this tutorial we are going to use the Python programming language which is actually very simple to program in even for a beginner (which I am). Now coming to  the image processing part, python alone is not capable of performing everything. We need some libraries that can handle image data and for that purpose I prefer using OpenCV which is an open source image processing and machine vision library.


So the first step, setting up the software which is a simple two step process (repeated thrice-Download>>Install).


Download Python 2.7.2.Install it (Nothing to say here I guess). Then download Numpy module for python 2.7. Remember python modules are version specific be sure to choose the module meant for your version of python In this discussion we are going to use Python 2.7. Numpy is a python module required for fast numerical manipulations on large arrays, image data in our case (more on Numpy soon). Download and install Numpy for python 2.7. 


After installing Numpy, download and install OpenCV 2.3. It is in the "opencv" folder on the local drive (C:\ in some cases). Copy the "cv.py" file from C:\opencv\modules\python\src2 into the python installation folder at the typical path C:\Python27\Lib\site-packages\ . Once done, you should have the stage set for image processing in python.


Image processing in python is simple when compared to that in other popular languages like C ,C++, Java. I prefer python over the others as its very simple to understand and is programmer friendly unlike C & C++.


Assuming you know how to program in python (if not visit my previous post ).


Start up the Python IDLE from the desktop icon. You get a command shell, this is the python shell where you can enter commands just like in Matlab (or command prompt).The commands entered here are executed immediately.


before using the image processing functions, we need to import the opencv module that we previously installed and copied. This brings all the functions encased in the opencv module into scope. 


("#" usually preceeds a comment in Python)
imports cv module into the python environment
>>>import cv


The cv.Load("file path") function is used to load an image in python. Use '\\' instead of '\' to ('\\' is python escape character for  '\'). This function loads the image data into the variable 'img'
>>>img=cv.LoadImage("C:\\Users\\Public\\Pictures\\Sample Pictures\\Koala.jpg")


To show that image, we need a window.So we create a window first using a function NamedWindow("window name",mode).mode=0(Auto Resize) mode=1(normal)
>>>cv.NamedWindow("Hello World",0) 


Tip: While typing a command hit 'Ctrl + Space' to see an auto completion box. It gives a list of all the function and modules loaded. Makes a programmer's life(very precious) heaven.


So we have created a window. Now to show the image in it. For that we better use a function called ShowImage("window name",image_variable)."window name" is the name of the window that we just created ("Hello World") and the image_variable is img in our case.


But the problem here is that the window we created has to be refreshed often using the ShowImage function. So we create an infinite loop that displays an image until the user hits 'Escape'. We use a WaitKey function to take input from the user. This is also a part of the opencv module. It makes the program wait for a  time defined in the parenthesis (in ms) before continuing to execute the code that follows it.
## infinite loop
>>>while True: 
        cv.ShowImage("Hello World",img)
        k=cv.WaitKey(10) #Waits for 10ms 
        if k==27:
             break


Here the image is refreshed ever 10ms. The first line is the typical while statement with the 'True' condition making it execute infinitely.The cv.ShowImage() refreshes the "Hello World" window with the image 'img'. The cv.WaitKey() waits for 10ms for a user input an returns the Ascii value of the input. It is copied into 'k' and if the value of 'k' is 27 (Ascii value of 'Escape key') the once infinite while loop breaks and the program  ends.


Note: The program ends but the window will not close as python doesn't do that by default. We need to add one more command at the end to destroy the window we created cv.DestroyAllWindows(). Use this command to close the  window instead of forcibly trying to close it. Always enter this command at the end of the program
>>>cv.DestroyAllWindows()


Tip: Python doesn't require a semicolon(;) unlike in C. But, you may use it to write and separate multiple statements in a single line.


You may enter all the commands into a new .py file (Hit 'Ctrl + N' in python window to open a python editor). Save and run the file to execute all commands sequentially.
Remember to put in the cv.DestroyAllWindows() at the end.

Python Editor with all the commands and the resulting window

I know its boring to read and display an image on the hardrive. Hence, make sure you have a camera attached to the PC (inbuilt camera will work in case of Laptops). Make sure their drivers are installed and the camera is working properly before running the program.

Download Amcap which is a small application to test and configure your camera settings.

Once we have your camera working, we need to access it from python. OpenCV library once again comes to our rescue by providing a function to access the camera, cv.CaptureFromCAM(camera_id) function. It takes the camera hardware id number, the numbering of camera hardware starts with the inbuilt camera being numbered '0' and so on.It returns a capture object through which we may capture frames from the camera.

>>>capture=cv.CaptureFromCAM(0) # '0' For the default inbuilt Camera

The above function not only creates an capture object but also turns on the camera.Now to capture image frames from the camera, we have a function called cv.QueryFrame(capture_obj). It takes in a capture object associated with a camera and returns an image.

>>>img=cv.QueryFrame(capture)
>>>cv.ShowImage("cam preview",img) #to show the image.


Note: Sometimes when the camera fails, the capture object can't be created and hence the cv.CaptureFromCAM() returns a '0'. The program will rise an error in that case.So we have made sure that the images are queried iff the capture object is created successfully (capture !=0) 


The overall code would be:

import cv

capture=cv.CaptureFromCAM(0)

while capture!=0:

    img=cv.QueryFrame(capture)
    cv.ShowImage("cam",img)
    k=cv.WaitKey(100)
    if k==27:
        break


cv.DestroyAllWindows()

I am tired. I guess you too are drooling over the keyboard.

If not, Thank You. That's it for now. This happens to be PART 1 of the Tutorial post.
See you in PART 2.

Do comment regarding any additional info needed. A 'like' on this page would definitely make me happy

Getting Started with Python: Happy New Year 2012

Let's start this new year by learning a new programming language. Okay, now that was a very geeky thing to say but who cares when it would add weight to your CV. Anyways, Python is an interpreted, Object-oriented and interactive language which is very simple to start off with. Even one who has little knowledge of computer programming may learn about it through this language.
 I planned on making some tutorials explaining python. But a lot of tutorials were already available,so why take the pain. There are a huge load of such tutorials and I have put here the best collection.
Before starting off, you need the following :


1.Python Software from here. Required to create and run the python programs.
2. PyScripter which is a free Intergrated Development Environment for Python


The first  program is enough to begin programming, while the second makes things a breeze  while you cruise through programming. PyScripter requires the Python Software to be installed prior. So, make sure it's there before programming in PyScripter.


The below tutorials make use of the above programs to teach you python (from Khan Academy)

Introduction to Programs Data Types and Variables

Python Lists

For Loops in Python

While Loops in Python

Fun with Strings

All the videos above are from Khan Academy. A lot more about python are available here

THIS POST HAS BEEN MADE TO SET THE MOOD FOR UP COMING ARTICLES ON MACHINE VISION USING PYTHON.