Skip to content

A tokenizer for the python programming language in C

Notifications You must be signed in to change notification settings

aymanizz/python-tokenizer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A Python Tokenizer

A tokenizer for the python programming language in C.

Differences

There are a couple of differences between this tokenizer and what cpython tokenizer does.

This tokenizer:

  • does not emit NL.
  • does not recognize encoding comment.
  • does not support unicode.
  • does emit tokens for each reserved keyword.

I plan on adding support for some of these but not all. The implementation will differ from the cpython's in many ways.

Building And Testing

  1. Clone the repository and initialize the submodules.
    git clone --recurse-submodules https://github.com/aymanizz/python-tokenizer.git
    
  2. Run make. The following make commands (targets) are available:
    • build the tokenizer: make.
    • build the tests runner and run it: make test.
    • remove the binaries directory: make clean.

Ouput executable files can be found in bin directory after building.

About

A tokenizer for the python programming language in C

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published