Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Real-Time Dynamic Programming #49

Merged
merged 32 commits into from
Aug 25, 2024
Merged

Real-Time Dynamic Programming #49

merged 32 commits into from
Aug 25, 2024

Commits on Apr 29, 2024

  1. Draft monte carlo value iteration 

    No glue whether the algorithm is called like that. I do value iteration
but chose the order of state-value updates randomly, weighted by current
state value estimates, the estimated optimal policy and an exploration
term.
    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    16baa30 View commit details
    Browse the repository at this point in the history

  2. Record statistics about how often states are visited

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    b6b6318 View commit details
    Browse the repository at this point in the history

  3. Guide exploration along honest policy

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    329b488 View commit details
    Browse the repository at this point in the history

  4. Clearly separate PTO from the solving algorithm

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    cc63083 View commit details
    Browse the repository at this point in the history

  5. Warm the agent up with honest policy

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    a5ccd8c View commit details
    Browse the repository at this point in the history

  6. Memoization

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    37207f3 View commit details
    Browse the repository at this point in the history

  7. Use "Exploring Starts" method in 50% of the episodes played

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    3b19ceb View commit details
    Browse the repository at this point in the history

  8. Be smart about choosing random start states.

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e0671d6 View commit details
    Browse the repository at this point in the history

  9. Store state hashes, not full states

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    ec68ba7 View commit details
    Browse the repository at this point in the history

  10. Speed up sm.py state editor. 

    Use pickle instead of numpy adjacency matrix. Adjacency matrix based
load save was O(n^2) in memory and time!
    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    bfab49d View commit details
    Browse the repository at this point in the history

  11. Take progress from defender chain instead of common chain. 

    We can now handle non-truncated models with monte carlo value iteration!
    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    5268dab View commit details
    Browse the repository at this point in the history

  12. Implement fair shutdown for sm.py

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    be20d16 View commit details
    Browse the repository at this point in the history

  13. Use fair shutdown for initial state value estimates

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    558103e View commit details
    Browse the repository at this point in the history

  14. Calculate expected future progress during optimization

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    80906d6 View commit details
    Browse the repository at this point in the history

  15. Add missing dependency

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    49d6591 View commit details
    Browse the repository at this point in the history

  16. It's called real time dynamic programming, I think.

    @pkel
    pkel committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    9e7e90b View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Draft RTDP evaluation script

    @pkel
    pkel committed May 6, 2024
    Configuration menu
    Copy the full SHA
    13c051d View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. compare rtdp start values to vi start values instead of steady state … 

    …values
    @pkel
    pkel committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b1b55ef View commit details
    Browse the repository at this point in the history

  2. add rtdp generic experiment; reveal bug in sm.SelfishMining.honest()

    @pkel
    pkel committed May 8, 2024
    Configuration menu
    Copy the full SHA
    74b9f62 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Fix honest action in generic model

    @pkel
    pkel committed May 9, 2024
    Configuration menu
    Copy the full SHA
    478c48c View commit details
    Browse the repository at this point in the history

  2. Extract working MDP from RTDP agent; report size of policy induced MC

    @pkel
    pkel committed May 9, 2024
    Configuration menu
    Copy the full SHA
    4485d42 View commit details
    Browse the repository at this point in the history

  3. Add todo note

    @pkel
    pkel committed May 9, 2024
    Configuration menu
    Copy the full SHA
    6cae85a View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Try analysing steady states; fail

    @pkel
    pkel committed May 10, 2024
    Configuration menu
    Copy the full SHA
    7fee1ef View commit details
    Browse the repository at this point in the history

  2. remove steady state related stuff

    @pkel
    pkel committed May 10, 2024
    Configuration menu
    Copy the full SHA
    2ad88c8 View commit details
    Browse the repository at this point in the history

  3. rethink exploring starts in rtdp

    @pkel
    pkel committed May 10, 2024
    Configuration menu
    Copy the full SHA
    b3d63f1 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2024

  1. improve exploring starts logic

    @pkel
    pkel committed May 12, 2024
    Configuration menu
    Copy the full SHA
    1f0fbb4 View commit details
    Browse the repository at this point in the history

  2. Revise shutdown and initial value estimate

    @pkel
    pkel committed May 12, 2024
    Configuration menu
    Copy the full SHA
    837a97e View commit details
    Browse the repository at this point in the history

  3. add rtdp debug notebook

    @pkel
    pkel committed May 12, 2024
    Configuration menu
    Copy the full SHA
    9a9c407 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. Revise shutdown of aft20 model, add reminder for generic model

    @pkel
    pkel committed May 17, 2024
    Configuration menu
    Copy the full SHA
    7f0ecd5 View commit details
    Browse the repository at this point in the history

  2. run policy evaluation for rtdp measurements

    @pkel
    pkel committed May 17, 2024
    Configuration menu
    Copy the full SHA
    cd4a8ae View commit details
    Browse the repository at this point in the history

  3. Enumerate available actions in mdp.Compiler; 

    and make this apparent in the mdp.tab data type
    @pkel
    pkel committed May 17, 2024
    Configuration menu
    Copy the full SHA
    e130f72 View commit details
    Browse the repository at this point in the history

  4. Evaluate rtdp policy in tabular mdp

    @pkel
    pkel committed May 17, 2024
    Configuration menu
    Copy the full SHA
    7a6fba9 View commit details
    Browse the repository at this point in the history