-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Autogenerate keywords base on files #313
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
af62aaf
autogenerator keywords
yceballost 0af5210
add dot env to workflow
yceballost 555aad3
Update keywords-generator.yml
yceballost 04404d1
fixing workflow
yceballost 2d0623f
Update keywords-generator.yml
yceballost b8db527
fix job name
yceballost 9fcf2ce
remove keywords from contributing doc
yceballost 39c97a5
downgrade model
yceballost 9cf93a6
fix missing commit in workflow
yceballost 81c0cf8
improve propmting
yceballost f01d75a
avoid failing workflow when nothing to commit
yceballost 85285b8
Update keywords-generator.yml
yceballost c6df326
Update keywords-generator.yml
yceballost File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
import os | ||
import json | ||
import openai | ||
import subprocess | ||
from dotenv import load_dotenv | ||
|
||
# Load environment variables from the .env file | ||
load_dotenv() | ||
|
||
# Set up the OpenAI API key | ||
openai.api_key = os.getenv("OPENAI_API_KEY") | ||
|
||
# Path to the main folder (the "icons" folder) | ||
main_folder = "./icons" | ||
|
||
# JSON file where synonyms will be stored | ||
synonyms_json_file = "icons/icons-keywords.json" | ||
|
||
# Load the JSON file if it already exists | ||
if os.path.exists(synonyms_json_file): | ||
with open(synonyms_json_file, "r", encoding="utf-8") as f: | ||
synonyms_dictionary = json.load(f) | ||
else: | ||
synonyms_dictionary = {} | ||
|
||
# Remove common style indicators from the filename and return | ||
def preprocess_filename(filename): | ||
return filename.replace("-filled.svg", "").replace("-light.svg", "").replace("-regular.svg", "") | ||
|
||
# Function to recursively list all unique SVG filenames without their paths | ||
def list_concepts(folder): | ||
concepts = set() # Use a set to avoid duplicates | ||
for root, dirs, files in os.walk(folder): | ||
# Filter and preprocess SVG filenames before adding to the set | ||
for file in files: | ||
if file.endswith('.svg') and not file.startswith('.'): | ||
processed_file = preprocess_filename(file) | ||
concepts.add(processed_file) | ||
return list(concepts) # Convert set to list before returning | ||
|
||
# Function to generate synonyms using GPT | ||
def generate_synonyms(concept): | ||
prompt = f"Generate 12 synonyms for the concept '{concept}' mixing English, Spanish, Portuguese, and German (in this order). Return them as a plain list of words, without quotes, numbering, or separation by language. the order of the synonyms should be English, Spanish, Portuguese, and German. For example: alert lamp cross, warning light plus, signal illumination cross, luz de alarma cruz, luz de advertencia plus, iluminación de señal cruz, Alarmleuchte Kreuz, Warnlicht plus, Signalbeleuchtung Kreuz" | ||
|
||
response = openai.ChatCompletion.create( | ||
model="gpt-4", | ||
messages=[ | ||
{"role": "system", "content": "You are a helpful assistant."}, | ||
{"role": "user", "content": prompt} | ||
], | ||
max_tokens=100, | ||
temperature=0.7, | ||
) | ||
|
||
# Get the generated response | ||
raw_synonyms = response['choices'][0]['message']['content'].strip() | ||
|
||
# Clean the output, removing unnecessary characters and convert it to a list | ||
synonyms = [synonym.strip(' "[]') for synonym in raw_synonyms.split(',')] | ||
|
||
return synonyms | ||
|
||
# Get concepts from the icons folder | ||
concepts = set(list_concepts(main_folder)) # Convert to set for efficiency | ||
|
||
# Counter to know how many new concepts have been processed | ||
new_concepts = 0 | ||
|
||
# Remove entries in synonyms_dictionary that no longer have corresponding icons | ||
keys_to_remove = [key for key in synonyms_dictionary if key not in concepts] | ||
for key in keys_to_remove: | ||
del synonyms_dictionary[key] | ||
print(f"Concept removed: {key}") | ||
|
||
# Update the JSON file only with new concepts | ||
for concept in concepts: | ||
if concept not in synonyms_dictionary: | ||
print(f"Generating synonyms for: {concept}") | ||
try: | ||
# Generate synonyms using GPT | ||
generated_synonyms = generate_synonyms(concept) | ||
# Save the synonyms in the dictionary | ||
synonyms_dictionary[concept] = generated_synonyms | ||
new_concepts += 1 | ||
print(f"Concept generated: {concept} - Synonyms: {generated_synonyms}") | ||
except Exception as e: | ||
print(f"Error generating synonyms for {concept}: {e}") | ||
# In case of error, save generic synonyms to avoid leaving it empty | ||
synonyms_dictionary[concept] = ["synonym1", "synonym2", "synonym3"] | ||
print(f"Concept generated with generic synonyms: {concept}") | ||
|
||
# Only save if there are new concepts or if concepts have been removed | ||
if new_concepts > 0 or keys_to_remove: | ||
# Save the updated and alphabetically sorted dictionary in the JSON file | ||
with open(synonyms_json_file, "w", encoding="utf-8") as f: | ||
json.dump(synonyms_dictionary, f, ensure_ascii=False, indent=4, sort_keys=True) | ||
print(f"{new_concepts} new concepts have been generated.") | ||
print(f"{len(keys_to_remove)} obsolete concepts have been removed.") | ||
else: | ||
print("No new concepts or obsolete concepts found.") | ||
|
||
# Run Prettier to format the JSON file | ||
try: | ||
subprocess.run(["npx", "prettier", "--write", synonyms_json_file], check=True) | ||
print(f"The file {synonyms_json_file} has been formatted with Prettier.") | ||
except subprocess.CalledProcessError as e: | ||
print(f"Error formatting the file with Prettier: {e}") | ||
|
||
# Count the total number of concepts | ||
total_concepts = len(concepts) | ||
print(f"Total concepts processed: {total_concepts}") | ||
print("Process completed. The icons-keywords.json file has been updated and sorted alphabetically.") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
name: Keywords autogenerated | ||
|
||
on: | ||
workflow_dispatch: | ||
pull_request: | ||
branches: | ||
- production | ||
paths: | ||
- "icons/**" | ||
|
||
jobs: | ||
generate-keywords: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: Checkout repository | ||
uses: actions/checkout@v4 | ||
|
||
- name: Get branch name | ||
uses: rlespinasse/github-slug-action@v3.x | ||
|
||
- name: Set up Python | ||
uses: actions/setup-python@v2 | ||
with: | ||
python-version: "3.x" | ||
|
||
- name: Upgrade pip, setuptools, and wheel | ||
run: | | ||
pip install --upgrade pip setuptools wheel | ||
|
||
# Paso 3: Instalar dependencias de Python | ||
- name: Install dependencies | ||
run: | | ||
pip install openai python-dotenv --use-pep517 | ||
|
||
- name: Install prettier | ||
run: npm install -g prettier | ||
|
||
- name: Run keywords generator | ||
env: | ||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
working-directory: .github/keywords | ||
run: python3 keywords-generator.py | ||
|
||
- name: Check for changes | ||
id: git_status | ||
run: | | ||
git diff --exit-code || echo "changes" | ||
|
||
- name: Commit & Push in ${{ env.GITHUB_REF_SLUG_URL }} | ||
if: steps.git_status.outputs.result == 'changes' | ||
run: | | ||
git add . | ||
git config user.name "github-actions" | ||
git config user.email "github-actions[bot]@users.noreply.github.com" | ||
git commit -am "update changelog" | ||
git push origin ${{ env.GITHUB_REF_SLUG_URL }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
automatic sort