| Pages:
1
2 |
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
I'll also add a kind of "filter", so let's say when you write "Sodium" it might also show SODIUM hydroxide or SODIUM carbonate right now, I'll have to
add some kind of system so it automatically only shows the element SODIUM.
I'll make it freely available for everyone! ( if I didn't already mention that eariler in the thread. ) Plus I'll probably also make the source code
public, so if anybody wants to make a tweak they can do it.
|
|
|
Chemi Pharma
Hazard to Others
Posts: 351
Registered: 5-5-2016
Location: Latin America
Member Is Offline
Mood: Quarantined
|
|
Hi my friend.
I already replied your post asking for price of reagents but I think you've a very, very hard job on your hands. prices changes like Dollar and EUROs
quotation changes. They can change everyday and every time. Like some actors here in this Blog already have said it's important to know which country
do you want to deliver the goods cause the Customs specifications that's different country by country. It will be a difficult job to create something
like you're wiching for, but I'm really wish you a good luck!
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
Thank you!
I mean, SciFinder already has done this. What I am really trying to do is just make a "SciFinder" for suppliers that sell to private individuals, and
is completely free. But I will do my best to include shipping costs etc.
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
Right now I am working on adding suppliers in the U.S.
I am using this forums list of suppliers, but I do not know which ones you guys prefer.
Right now I have added Carolina Chemical, Science Company and Elemental Scientific. Which ones do you all think I should add next?
|
|
|
Texium
Administrator
Posts: 4642
Registered: 11-1-2014
Location: Salt Lake City
Member Is Offline
Mood: Preparing to defend myself (academically)
|
|
Well, Elemental Scientific is shutting down and clearing house, so you might actually have to remove them.
If you are willing to include suppliers that are not officially chemical suppliers but sell a good variety of chemicals, Duda Diesel and Seattle Pottery Supply are among the most popular.
As far as actual chemical suppliers go, Ambeed has basically everything, but while they will allow you to purchase with a personal credit card, I’m not sure if they ship to residential
addresses. The only time I ordered stuff for myself from them, I had it shipped to my university address just in case. So someone else will have to
confirm if they truly ship to individuals. Same goes for Oakwood Chemical, Combi-Blocks, and aablocks.
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
Ah, didn't know that - will remove Elemental Scientific now then.. It was actually so difficult to add Elemental Scientific :/
And yes, I can definitely still add Duda Diesel and Seattle Pottery Supply. I geuss amateur chemistry is all about getting chemicals from accessable
sources, so it makes sense to add them.
I am pretty sure Ambeed ships to residental addresses, so I'll add them too.
Thanks for the list!
|
|
|
chempyre235
Harmless
Posts: 28
Registered: 21-10-2024
Location: USA
Member Is Offline
|
|
I have compiled a list of potential suppliers in the US. Keep in mind that I have not purchased anything from these sites, and so may need some
vetting before adding them to your software. I tried to exclude sites already listed on this site. Also, I have not verified that all of these, sell
to residential addresses.
Some of these I think will be a good addition. There is one that specifically sells solvents and other chemicals relatively cheaply, and for amateur
purposes. There is also another site I found that sells polymer precursors. I hope it helps! 
Attachment: supply.docx (14kB)
This file has been downloaded 73 times
[Edited on 12/17/2024 by chempyre235]
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
That's a lot of suppliers chempyre235!
Thank you - it definitely helps!
|
|
|
Cathoderay
Hazard to Self
Posts: 59
Registered: 29-1-2023
Location: US-Texas
Member Is Offline
|
|
I have ordered many things over the years from McMaster-Carr as an individual. The shipping method is fast but pricey. The website is simply www.mcmaster.com
|
|
|
chempyre235
Harmless
Posts: 28
Registered: 21-10-2024
Location: USA
Member Is Offline
|
|
I found a couple more possible sites:
https://www.chemimpex.com/
https://www-live.chemdirect.com/
Again, they need checked because I haven't ordered from these.
|
|
|
kyfuge
Harmless
Posts: 2
Registered: 25-3-2024
Member Is Offline
Mood: Lurking
|
|
Quote: Originally posted by chempyre235  | I have compiled a list of potential suppliers in the US. Keep in mind that I have not purchased anything from these sites, and so may need some
vetting before adding them to your software. I tried to exclude sites already listed on this site. Also, I have not verified that all of these, sell
to residential addresses.
Some of these I think will be a good addition. There is one that specifically sells solvents and other chemicals relatively cheaply, and for amateur
purposes. There is also another site I found that sells polymer precursors. I hope it helps!
[Edited on 12/17/2024 by chempyre235] |
The list says Carolina Chemical doesn’t exist anymore, but their website still works for me. Carolina Biological is an entirely different company.
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
As I am soon done polishing the european version, if anybody want's to be a tester for my software, I'd appreciate it!
I'll still need to fix a few things.
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
It was a bit more dififcult to do in javascript, so I decided to do it in python. When I am done, I can always just do it in JS afterwards too.
Anyway, this is how it's looking in a console app so far. I am still adding the shipping, quantity, notes, etc.
I still need some beta-testers, if anybody is interested? I am still not done for the beta-version yet, so it's going to take a while still.
Also, it is a bit more difficult for elements. For example if you search "sodium", the top results are sodium chloride, etc, but I'll get to fixing
that.
[Edited on 9-2-2025 by Maui3]
|
|
|
SuperOxide
National Hazard
Posts: 509
Registered: 24-7-2019
Location: Devils Anus
Member Is Offline
|
|
Couple of questions,
- You said this will limit the scope to suppliers that sell to individuals. How about for specific products? eg: Carolina Chemical sells to
individuals, even 1L of H2SO4. But if you try to buy the 2.5L (or larger) bottles of H2SO4 then they require that you be a school or run a business
(and ship to those addresses)
- How does this pull the prices? Did you go to each individual supplier and see if the frontend sends requests to a backend API, then use those? Does
it just scrape the HTML for simpler websites? I'm hoping you have some simple modular system that allows for small changes (eg: if the paths or ID's
of the elements change, they use pagination, etc).
- Do the results get cached for any amount of time? eg: If the same SKU or CAS gets pulled up in different search results, it might not make sense to
query/parse the same web page/product multiple times within 24 hours).
- Is this going to be open source?
Hopefully you're structuring it so that there's just some logic module that does the data, then that can be used in either a CLI client (like what
you're showing) or use something like FastAPI to create a basic web client.
I'm a software engineer, I do NodeJS for my professional job but I do a bunch of Python on the side (mostly for 3D printing related, eg: moonraker),
and I'd love to see the code for this :-) (show me yours, ill show you mine).
|
|
|
Maui3
Hazard to Others
Posts: 128
Registered: 9-9-2024
Member Is Offline
|
|
Of course! I love you alls questions and feedback!
1: I have not checked the website for Carolina Chemical, but for some of the european ones there will be a little popup saying that it is not sold to
private individuals above a certain concentration or amount. My python script could potentially check for that.
2: This was very difficult to get to work, so the way it works right now is like this: It uses the search link, for example: "example.com/search=" +
the chemName. Then it uses beautifulsoup to extract the HTML, where it finds the product and price from their classname. As you noted, this can be
quite unreliable - so I am definitely also considering possibly doing a daily check for each of the websites in the sitemap.xml (since it'll take too
long to do for each individual search), whereafter it'll find the product name and price - possibly by finding the first currency-symbol and then the
closest chemName to the price-element. Of course chemical names differ, so it'll run through the Pubchem API for all of the synonyms.
3: That's kind of what I considered too (like I said in 2) - but so far it only checks for individual searches. There is not database, yet. I also
thought that if I was to do this, I would do it as one of the last things, since making it extract the price and chemical name is the most important
thing.
4: Yes, of course! Besides, if the chemical suppliers websites whole system and URLs change, I would much prefer if it could be fixed by others than
me. Also, I am just a amateur chemist and amateur programmer, it would be very ineffecient and ignorant to only let me edit it and understand how it
works.
Yes I am structuring it like that.
Oh, maybe you can help me with some of it then - I will definitely need help at some point.. probably more than just one point. If you would
like to help me with whereever I need help, I'd really appreciate it.
Yes - I'll show you the code. It's structured quite poorly now - with variables that are.. well.. badly named. I'll fix it and send it to you in just
a bit. Where (and how) would you prefer me sending it to you?
EDIT: I would loved to have used the suppliers APIs, I really would - but most of the suppliers never replied back to me after I told them I would
like an API. There was one that replied back saying they'll send my message to their "tech-team", after which they could send me the API. It's been
three months, they have no intentions on doing that.
[Edited on 9-2-2025 by Maui3]
|
|
|
SuperOxide
National Hazard
Posts: 509
Registered: 24-7-2019
Location: Devils Anus
Member Is Offline
|
|
| Quote: Originally posted by Maui3 | 1: I have not checked the website for Carolina Chemical, but for some of the european ones there will be a little popup saying that it is not sold to
private individuals above a certain concentration or amount. My python script could potentially check for that.
|
Carolina Chem is an easy one for this, it always just says "This item is sold and shipped to schools and businesses only.".
| Quote: Originally posted by Maui3 |
2: This was very difficult to get to work, so the way it works right now is like this: It uses the search link, for example: "example.com/search=" +
the chemName. Then it uses beautifulsoup to extract the HTML, where it finds the product and price from their classname. As you noted, this can be
quite unreliable - so I am definitely also considering possibly doing a daily check for each of the websites in the sitemap.xml (since it'll take too
long to do for each individual search), whereafter it'll find the product name and price - possibly by finding the first currency-symbol and then the
closest chemName to the price-element. Of course chemical names differ, so it'll run through the Pubchem API for all of the synonyms.
|
I've had to automate stuff like that. It's... even less fun than it sounds, lol.
A lot of these websites keep the chemical synonyms paired with the listing, so typically searching for any of them should return the same product (at
least on a few that I tried).
| Quote: Originally posted by Maui3 |
EDIT: I would loved to have used the suppliers APIs, I really would - but most of the suppliers never replied back to me after I told them I would
like an API. There was one that replied back saying they'll send my message to their "tech-team", after which they could send me the API. It's been
three months, they have no intentions on doing that. |
Well, sir, since it would be a client side API (if its used on their website), just keep an eye on your networking tab of your browser.
Example of how to do it with LigmaAldrich
1) ProductSearch query:
gets a response with unique product keys:
2) Using the unique productKeys, request the Pricing and Availability endpoint:
Which will give you the price:
I think this would almost have to be done for some of them that populate the UI using RESTful calls from the client side. Otherwise, whats the best
option? Launch a headless browser to track the client side requests? Way more tedious.
| Quote: Originally posted by Maui3 |
3: That's kind of what I considered too (like I said in 2) - but so far it only checks for individual searches. There is not database, yet. I also
thought that if I was to do this, I would do it as one of the last things, since making it extract the price and chemical name is the most important
thing.
|
I guess this comes down to if you want it to be a stand alone client, or a service you provide? (Or either?) If its a service you provide, I would
have it cache the prices only on the first time its requested, and have a TTL for the prices that can expire, and only then will new requests for that
item trigger actual price updates.
If its a stand alone client, then I would do the same thing, but store it in a sqlite database.
| Quote: Originally posted by Maui3 |
4: Yes, of course! Besides, if the chemical suppliers websites whole system and URLs change, I would much prefer if it could be fixed by others than
me. Also, I am just a amateur chemist and amateur programmer, it would be very ineffecient and ignorant to only let me edit it and understand how it
works. |
Excellent! Open source is the way to go.
This is something you really need to put some time into and think about. It looks like you're just developing a Proof Of Concept right now,
so its not a huge deal, but when you work on the first release this is something that is key. Every website operates differently, but you want the
same data. So I would create it in a way that each site has its own module that just takes an input, then it can go about getting the data (cache,
RESTful API search, HTTP requests and parsing data, etc), then return a simple object with the results. Obviously each one should be in its own file,
making it super easy to replace/update them when needed.
| Quote: Originally posted by Maui3 |
Yes - I'll show you the code. It's structured quite poorly now - with variables that are.. well.. badly named.
|
That's a bad sign, though I'm guilty of it as well. If you're doing this in Python I would highly recommend putting a hold on your coding and read
through the PEP8 style guide first. It seems silly, but coding in a specific style will save you headaches later on. And its just more professional, so you
should get use to it ;-)
| Quote: Originally posted by Maui3 | | I'll fix it and send it to you in just a bit. Where (and how) would you prefer me sending it to you? |
Then throw this baby up on Github, I'd love to take a look at it :-) I do a lot of programming for work, so I rarely dive into more programming on the
side, but this might be a good way to contribute to the amateur chemistry community that has helped me so much.
[Edited on 10-2-2025 by SuperOxide]
[Edited on 10-2-2025 by SuperOxide]
|
|
|
SuperOxide
National Hazard
Posts: 509
Registered: 24-7-2019
Location: Devils Anus
Member Is Offline
|
|
Here's an example of how to use Chemsavers API endpoints.
Sending the query for toluene:
And you get back a pretty beefy response with all the related products, and their prices:
But how do you do it programmatically?
Grab the curl request command for that specific request:
Throw that in your console, add some jq to make it pretty and get the data you want:
Do you need it in a different language? Python you said? Use CurlConverter.com for that:
The same process would work for AmBeed:
Obviously it might be better to search by CAS for Ambeed, or you could just look through the results to make sure they're what you expect. I searched
for "toluene" and got a lot of results, none of which were actually toluene.
Elementalscientific has something that you can query, it returns the price as HTML, but that makes parsing it a heck of a lot easier:
Before you go through and do all the work on the search code for your tool, I would say go through to each chemical supplier and see if they have a
public facing API when you search on their website. Create a list of which ones do have API's, which don't, and which ones are a hybrid (some have an
API for searches, then you'd need to parse the HTML for the price - example: Carolina Chemical).
[Edited on 10-2-2025 by SuperOxide]
|
|
|
| Pages:
1
2 |
![[*]](images/xpblue/default_icon.gif){kind=icon}
