Feature #3607
Implementing AI into Hope project
| Status: | Assigned | Start date: | 03/04/2024 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | 04/30/2024 | |
| Assignee: | Kasthuri G R | % Done: | 0% | |
| Category: | - | Spent time: | - | |
| Target version: | - |
Description
Take input words from Hope, generate sentences using large language models (LLMs) with basic commonly used words
History
#1 Updated by Kasthuri G R 8 months ago
Google has released an open AI model Gemma(https://blog.google/technology/developers/gemma-open-models/). I am able to generate sentences for the given input words.
I have provided the GSheet link below with the input words and corresponding AI results generated by Gemma model
[[https://docs.google.com/spreadsheets/d/1CluoU2D38b6nuRMt4AJDTpU6_GFoB0e3VcQOo0_QJ0Y/edit?usp=sharing]]
This is open source model and it gives expected results. We will have to design the prompts to get better results based on the user needs. I will set a deadline based on the list of features we deciding
#2 Updated by Kasthuri G R 8 months ago
To get inputs from special educators, made a sample UI in python that user can upload image and it will generate sentences for the given image. I have implemented the functionality. Yet to make the UI better with background color and alignment.
#3 Updated by Kasthuri G R 8 months ago
Made a sample UI that allows to give input image and generate sentence. The model output varies like below. I am trying to figure out the method to filter the output.
For eg: Given Prompt "construct 5 basic proper sentences about {input word} and its habits for primary school students"
AI output
Step 1/5
1. Parrots are beautiful birds that come in many different colors and sizes.
Step 2/5
2. Parrots are very smart and can learn many tricks and even talk.
Step 3/5
3. Parrots are social birds and love to be around other parrots and people.
Step 4/5
4. Parrots need a lot of attention and exercise to stay healthy.
#4 Updated by Anil Prabhakar 8 months ago
- Tracker changed from Bug to Feature
- Project changed from CREATE to AAC
- Due date set to 04/30/2024
- Status changed from New to Assigned
There should be a Due Date for every task and an updated %Done. Cannot work without any deadlines.
- Do we have a creative common license to use these images?
- How are you imagining the use case, is this set up by a teacher, and the child clicks on a button, or is the child supposed to upload an image?
- This is not picture to speech (think AI into HOPE...as the subject of this Issue). I am expecting a communication tool, where a user selects 1 or more images from the KAVI-PTS library, AI generates a few possible sentences, and the child picks one of them.
#5 Updated by Kasthuri G R 8 months ago
- File ThankYou.png added
- File Classroom.png added
- File Nice_to_meet.png added
- File Sorry.png added
Do we have a creative common license to use these images?
I will use the KAVI Library images
How are you imagining the use case, is this set up by a teacher, and the child clicks on a button, or is the child supposed to upload an image?
This is a sample UI we decided to create and show it to special educators in Mira school to get their ideas on the requirement. So we can plan the development in both Hope and Kavi-PTS based on their feedback
The sample use cases I have considered are below
1. AI to generate Sentences with the "Thank You, Sorry, Nice to meet you" which children can use. These are the pictures under Quick talk in the Kavi Library
2. Common sentences used by the children in Classroom. Class room is also the picture from our image Library
The respective AI responses for the above inputs are uploaded.
#6 Updated by Anil Prabhakar 8 months ago
- Use the > to quote, otherwise your reply is mixed with my comment.
- Sentences are quite ambiguous and will become frustrating. We may need more than one image to create a sentence. Let us try with 2 images.
#7 Updated by Kasthuri G R 6 months ago
When tried to run the python code where it takes input as Image(one) and generate text. It didn't run as huggingface has updated the libraries and the code need to updated to run the model without error. I have fixed the library issue and made the code functional.
I will modify the code to accept two images as inputs to generate text tomorrow
#8 Updated by Kasthuri G R 6 months ago
- File 1.keywords-dog-play-ball-happy.png added
- File 2.keywords-dog-play-ball.png added
- File 3.keywords-I-sleepy-hungry.png added
- File 4.keywords-I-pain-sad.png added
- File 5.keywords-sorry-helpme.png added
- File 6.keywords-we-sorry.png added
I made a UI to display the images. Images can be select/deselect on clicking. Once selecting the images, Clicking on Generate sentences will display the possible sentences with the keywords(image names).
I am sharing the sample outputs of the sentences generated by gemma model.
Prompt used "Generate sentences suitable for children using the all the key words. Make the sentences short, simple, and conversing. ";
Need to try with different prompts and have to run different models to compare the results
#9 Updated by Anil Prabhakar 6 months ago
Think about your conversation with anyone. When you are trying to communicate with your son, will the prompt be "generate suitable sentence"?
Don't be a programmer, put more thought into how you will use GenAI to build an AAC solution.