Project Mickey

goodbye

2006-01-19T10:30:00.000-08:00

Well, the CS30 project worked: we now have a Java program that can convert the logs into records in the new database. Dr. Iba says they just need to move the database to another account besides mine, translate some of the check queries so they will work with the new database, and of course there is more to do like making a form where the data can be edited, and a way to analyze the data as well as enter it. However, the database is very very close to being usable, so my part is pretty much done. If anyone has any questions or anything I'm of course available. I still have the notes that I've made as I was designing it. So keep in touch if you want. I've learned a lot... thanks, all.

goodbye

significant progress

2005-11-16T16:44:00.000-08:00

So I was waiting outside the DC and I caught sight of Dr. McMahon as she was going to a meeting or something. And all of a sudden I realized that while I'm not making hardly any progress on the virtual parasite project part of my research this semester, I can at the very least finish Project Mickey (which actually would be useful to them, like, NOW).

So I sat down and:

* Finished the php for the Record Death page
* Changed the mouse mark to a selection box instead of a text field
* Reloaded all of the database tables so that there are two fields for the entry date and entry time instead of just one timestamp
* Changed the php files so that they send two values: one for entry date and one for entry time instead of just one timestamp
* Fixed the Java Script which checks to make sure that every field of the form contains something (excluding the notes field, of course, which is optional). I even figured out how to check selection boxes which had been giving me trouble before.
* Tested everything
* Found some problems
* Fixed them.

WOOT!

Dr. Iba and I worked out how we could use a java program to go through the logs of the data that has already been entered. If it can output a string that is a correctly formatted URL, that can be sent to the php file which will format an SQL query with will insert it into the new database (since that is what the mouse entry forms do anyway). It would probably use the string parser that our CS30 class created for the CS130 project. It'll run through the file three times.

The first time, it will find all the records that include birthdates for mice. It will create an entry in the mice table, and two records in the moves table: not-born to birthcage, birthcage to weaned cage. This first time, the transfer date for BR to W will be the same as the birthdate.

The second pass through it will find all the records that include cage transfers and it will create an appropriate entry in the moves table.

The third pass through it will create all the records for the readings table.

Then we can go back and change the transfer date for BR to W for each mouse to the date of the first reading for that mouse. There's no guarantee that they didn't make a couple readings before recording a mouse's birthdate, so we have to do this last.

Dr. Iba will probably make this a CS30 project which is really cool--it'll be interesting to work on a CS30 project that I really care about because it's MY project. I hope the rest of the class doesn't hate me for making work for them though ;-)

Between now and then though Dr. Iba is going to have some of Dr. McMahon's students test-drive my entry forms. It's so nice to be able to make actual progress on something for a change.

Blog Maintenance

2005-07-05T01:06:00.000-07:00

I made a couple of executive decisions today about this blog. I'm not particularly fond of HaloScan comments now that Blogger revamped theirs. So since there was only one comment, I copied it, switched to blogger comments and reposted it. I also set it up so that when you post a comment it will automatically e-mail it too me (thus making this blog a great way to keep in touch with me about Project Mickey stuff as I work on it).

Also, since it was way easier to just reload the template for this page instead of sifting through the html to remove the HaloScan comments, I decided to update the links we had on the side bar since they didn't have a whole lot to do with what we were doing anymore. The html and css tutorials are condensed into one link to the main tutorial site, the CS research conference (that we didn't end up going to) is deleted, and the link to POV-Ray graphics... umm... we're not doing graphics anyway.

Not sure how much time I'll have this week, but I tend to only work on my other research project during the day, which leaves a ton of time in the evenings and nights which I usually spend on my computer anyway. Might as well be productive if I'm not too wiped :-)

New Mouse Pages Done

2005-07-02T12:58:00.000-07:00

Note to self: ctrl-x-s is the shortcut for saving when emacs menus don't work

I eased myself back into really working on this today--if I'm not careful I get scared to start. But it's going really really well. I wrote queries for entering a new mouse, and then I wrote the php and set up the page that displays the results so you can see what you entered. What was really cool was there were very few errors (like missing quote marks) that I had to fix. It was a lot easier than some of the first pages were.

Unfortunately user errors during entering aren't as easy to fix as entering a reading twice or something--you can't just delete everything and re-enter it. So I included instructions in case of errors: contact your local database guru. lol. I'm not sure how to take care of these kind of errors as a more permanent solution. In order to set everything right a person really needs to know how the database is set up and the actual queries that the php sends. It's kind of a bummer, but any non-CS people who use this will probably need to keep a CS person on hand. At least I can wire in enough that the CS person doesn't have to be called on for a every single thing, but... well, I guess being needed is handy ;-)

I only have Cage Transfer and Record Death pages left. I'm not going to tackle the view full record until I get back to Westmont. It should be pretty easy--the new mouse page already includes the query I'll need for both transfers and deaths, so there can be a fair amount of copy and paste. (I wish I could do it tonight and get it all done, but I've already been working on this for like 9 hours today, and it's almost midnight). After that is done though, comes the most difficult part: figuring out how to add everything to the database so that Dr. McMahon can start using them. I'm working on it, but I think I need to brainstorm with Dr. Iba a bit.

Update from Virginia

2005-07-01T17:35:00.000-07:00

WOOOOOT!!! (<-- always a good sound to hear) I am in Virginia, but I have sucessfully managed to ssh to wardrobe AND get the project mickey pages to display on my browser. MAJOR break through. Thanks for the instructions Dr. Iba :-) I've done a bit of blog maintenance, switching things around so that both of my research blogs can be accessed through my profile --> I'll be keeping this blog up as I continue on Project Mickey--not so much as a formal scientific logbook, but as a place to record my progress, thoughts, frustrations, questions, etc.

I don't guarantee that I will finish this quickly (given that I have a lot to work on with the internship here) but I feel bad that I wasn't able to finish it in time, so I'm going to try to work on it at least some (especially since I have the whole weekend ahead of me). The problem is, even if I finish the web pages, there will need to be some major data entry before it could be used. In order to enter a reading or anything, the code goes and searches for the mouse in its database. If it doesn't find it, then the reading can't really be entered. Not only does each mouse need to be entered into the mice table, but since the code searches for it based on its current cage, all of the cage transfers for the mice are going to need to be entered also. I know that some information was added by Julia last year, but I'm not sure how to add it to this new database given the new schema. NOT a pretty problem :-(

Once I get the code done, and especially when the database is up and running, a bit of code will need to be added so that every entry into the database is logged in a text file. That way if anything happens, we can rebuild the database easily. Especially since yours truly can go in and seriously mess things up in only a few key strokes. the power of the password...

Student Research Symposium Preparation

2005-04-16T16:51:00.000-07:00

Cailin and I decided, with Dr. Iba's suggestion, to participate in the Westmont Student Research Symposium. We still have a lot of work to create a working example of our database program, but are hopeful that data will be submitted to the database and basic retrieval will be available.

As the year draws to a close, I have been considering future work that will be required in this project. First, the software program must be completed. The "advanced analysis" that we state will be included in the program has yet to be developed. Second, in light of the larger area of investigation - how computing tools can be used to further biological research - it will be very important for us to analyze what was helpful about our project and what areas need improvement.

To Do

2005-03-28T16:05:00.000-08:00

* create diagram of database relations
* create the tables based on the diagram
* figure out how to find a mouse
* [populate the new tables with the old data]
* write query to find MID using wean cage and mark
* write php so that new data can be inserted from html pages
* put web pages on server

PHP Breakthrough!

2005-02-14T15:05:00.000-08:00

Annie and I met with Dr. Iba on 2-14-05.

Ayoub's Data
We practiced creating a view in postgresql that combined two tables.

Web interface
Since we ran into some problems making perl connect the database with the web interface, we decided to use PHP instead. Dr. Iba and I continued after Annie left for work and we succeeded, with the help of another student's example, in inserting data from an html form into a sample table. The next step is to make separate PHP files for each web page (similar to what we did for the perl files), with a function to insert data into the table when they hit submit. Once this is done we will be able to get the web interface up and running so that Dr. McMahon can use it to input data. Dr. Iba also showed me how to access my files on wardrobe, and I will be looking into transferring files from my computer into that directory using PSCP.

Meeting with Julia

2005-02-02T14:09:00.000-08:00

We met with Julia today and did some basic searches, like how many males and females are there of:
all the mice
all the mice with one paw of >= 1
all the mice with one paw of >= 2
all the mice with one paw of >= 3
all the mice with one paw of >= 4

Next problem is how to find all the mice where any two paws have a given score.

Developments Since the Last Post

2005-01-31T16:31:00.000-08:00

PostgreSQL:
The last week or so we have been working on switching the database over to PostgreSQL, as well as learning to log on, open the database, and run queries.

Perl:
Annie and I worked on finding a good template for the perl script which communicates between the HTML form and the database. Dr. Iba gave us a couple possibilities online, but we weren't sure what to do with them or it they would work, and we couldn't find any useful ones of our own. In the meeting with Dr. Iba today we discussed it and then Dr. Iba worked on it some, but ran into some problems. In the meantime Annie and I are supposed to modify the perl script we had for the first New Readings page to work for the other HTML pages. Although the script just writes the form data to a text file, it is a good place to start until it can be made to communicate with the database.

Julia:
Annie and I are setting up a meeting with Julia this week. Dr. Iba wants Julia to come by so that they can work on database corrections.

Change to Postgress

2005-01-22T16:30:00.000-08:00

Cailin and I met with Dr. Iba on January 18th. We decided to switch from MySQL to Postgress. We also did some practice SQL queries.

First Meeting with Julia

2005-01-22T16:19:00.000-08:00

Cailin and I met with Julia for the first time on January 21st. The meeting had slightly disappointing results. Julia requested to correct errors that she had recorded making when she entered data. She also had three data analysis questions for us. Cailin and I were unable to access the database, so we took note of the tasks Julia wanted to accomplish and agreed to meet again soon.

Meeting with Everyone

2005-01-22T16:14:00.000-08:00

On January 14th, Cailin and I met with Dr. Iba, Dr. McMahon and Julia. We discussed the progress that Dr. McMahon had made with Dr. Iba in cleaning up all of the data. They spent several hours checking that all the data appeared to be correct and found numerous mistakes. They also did a bit of preliminary analysis of the data. We discussed the idea of researching data cleansing techniques that would make this process much easier in the future.

Julia, Cailin and I agreed to meet on January 21st to begin querying the database.

Database Connection Language

2005-01-13T21:53:00.000-08:00

Annie and I met on 1-13-05 to discuss the pros and cons of various languages for connecting the webpage to the database. After some internet research we decided that JSP was probably not really what we wanted. While Perl looked more difficult than PHP it looked more powerful and less limited than PHP. Thus we chose Perl.

The research proposal was postponed to tomorrow after the meeting with Dr. Iba, Dr. McMahon, and Julia.

Interface Tweaking

2005-01-11T21:56:00.000-08:00

Annie and I met on 1-11-05 to accomplish the tasks on our to do list for the week.

We looked at java script first. As it turned out the 'or' operator was incorrect some how, so she just set the problem scripts to check and make sure that none of the fields equaled zero, instead of limiting the values to specific lengths. This made the code a little messier, but it is now bug free.

After checking with a reference, linking the pages turned out to be very easy, so we modified the URLs in the html and tested it. The pages are now linked.

We chose the Enter a Reading page to test the Perl script modifications, and changed the fields and number of fields as noted in the Perl script file. We e-mailed it to Dr. Iba to check.

The meeting with Dr. McMahon was set for Friday afternoon, and Annie and I decided to meet Thursday evening to go over the research credit application.

New Semester Level Setting

2005-01-11T21:44:00.000-08:00

We met with Dr. Iba on 1-10-05.

Since Annie and I will be doing this project for C.S. research credits, we discussed how many units we should apply for (Dr. Iba said we should plan on doing 2-3 hours per week, per unit). If we get 3-4 units for this project (and semesters can be combined) this research will count as one C.S. elective.
We also discussed goals for the semester, and they are:
--To provide an interface that allows for data entry and minimal data analysis.
--To learn SQL so that both Annie and I can run searches on the database that are requested by Dr. McMahon or Julia.
--To work out a way to export data from the database so that it can be graphed using Excel.

Dr. Iba described what he had done over Christmas break on the schema and some of what he and Dr. McMahon had discussed.

We also created a to do list for the week:
--Link the html pages together
--Fix the bugs in the java script
--Modify the current Perl script file to fit one of our html pages
--Set a date to meet with Dr. McMahon for more level setting

Schema

2005-01-11T21:36:00.000-08:00

On 12-9-04 we met with Dr. Iba.

We decided to create a Schema to define tables that we could reload as necessary. We experimented with different programs to create it and ended up with an initial draft of it. Dr. Iba said he would complete it over Christmas break.

The java script for the web interface is still not working, and the html pages need to be linked together.

Basic Database Design

2004-12-09T01:14:00.000-08:00

At our meeting with Dr. Iba on 12-6-04, he introduced us to basic database design.
We learned that the purpose of a database is to model data, and thus the first step is to determine what exactly we are modeling, and then to break that up into pieces: entity sets (shown by boxes) and relationships (shown by diamonds) between them. Each may have attributes (shown by boxes with rounded corners) which usually include a primary key (underlined) that identifies a tuple uniquely.
We arranged the sets and relationships, thinking about what needed to relate to what and how:

Entity Relationship Diagram

At first we were thinking that the date would be part of the attributes of Readings and Move Mouse, but decided to make Date a separate entity set. Comments likewise was going to be an attribute of Move Mouse, Date of Death (DOD), and Make Reading, but since this would require storing large amounts of data in relationship tables, we decided to make Comments a separate entity also.
We considered how to represent a mouse death. Perhaps using a Move Mouse where the cage it is moved to is called "dead." The date and comments associated with this move would have to be specially identified such that they would display not just as a cage transfer but as Date of Death (DOD) and Comments (type: death).
We realized that there might be a problem if two people enter two readings or moves at the exact same time with the same data. We tabled the problem until a later date.

The next task was to define relations (tables) based on the attributes. We decided to use MySQL, and were given accounts on the CS department server.
As of 12-8-04 both Annie and I have succeeded in downloading the necessary software and logging on to our accounts and MySQL. Annie also found the table Dr. Iba created. The tables still need to be created.

Summary of Work to Date on Project Mickey

2004-12-08T16:09:00.000-08:00

This Post summarizes our work on Project Mickey from its creation to present.

First Meeting with Dr. Iba:
Dr. Iba announces at the Computer Science dinner that there is a research opportunity involving databases and work with the Biology Department. Cailin and I (Annie) were very interested and talked with Dr. Iba. After learning about the project, Cailin and I excitedly discussed the potential of the project.

First Meeting with Dr. McMahon:
Cailin and I schedule a meeting to learn more about the project, by this time we're very excited about it! Dr.McMahon shares with us about her project. In a nutshell, she explains that she is studying arthritic mice. She has been tracking the progress of arthritis in each paw of hundreds of mice. She hopes to convert her thick notebook of mouse readings into an organized computer format that will allow her to generate graphs and charts for both pattern analysis and presentation.

Following our first meeting with Dr.McMahon, we meet with Dr.Iba to discuss our role in creating this program. We learn that the database will have an HTML user interface. Cailin and I return to our room and frantically begin using every whiteboard we have to discuss possible designs.

Dr. McMahon gives a presentation:
Cailin and I attend Dr.McMahon's biology lecture on her past research projects. Both of us are amazed by the complexity of her research! After the lecture we present our mock-ups of the user-interface. We discuss changes and finalize the format.

Cailin writes the HTML:
Cailin undertakes the long task of learning HTML and creating all the pages we've designed. She amazes me with the finished product!

First Meeting with Julia:
Cailin and I show Julia and Dr.McMahon our somewhat complete HTML user-interface. Julia expresses her concern for needing to correct errors she makes in data input. I decide to create Java-Script code to perform checks on the data before it is submitted. Julia also shares ideas to make data input faster.

Meeting with Dr.Iba: