Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Qdrant VectorDB #812

Closed
wants to merge 88 commits into from
Closed

Add support for Qdrant VectorDB #812

wants to merge 88 commits into from

Conversation

hkhajgiwale
Copy link
Contributor

@hkhajgiwale hkhajgiwale commented May 31, 2024 •
edited by github-actions bot
Loading

Description

This pull request introduces a new feature that integrates Griptape with Qdrant VectorDB, enhancing the capabilities of Griptape for handling large-scale vector data.

Problem Statement

Managing and querying large-scale vector data efficiently has been a challenge. The current solutions lack seamless integration for ingesting, processing, and indexing vector embeddings, leading to inefficiencies in AI and machine learning workflows.

Solution

The integration between Griptape and Qdrant VectorDB provides a scalable and efficient solution for ingesting, processing, and indexing vector embeddings. Leveraging Griptape's powerful data pipeline capabilities and Qdrant's advanced vector search technology, this integration supports a wide range of AI and machine learning applications.

Key Features

  • Seamless Ingestion: Efficiently ingest large-scale vector data into the system.
  • Advanced Processing: Utilize Griptape's robust data processing pipelines to handle vector data.
  • Efficient Indexing: Index vector embeddings using Qdrant VectorDB for optimized query performance.
  • Scalability: Support for large-scale data applications ensuring performance and reliability.

Alternatives Considered

  1. Using standalone solutions for data ingestion and vector search without integration.
  2. Employing other vector databases that might not provide the same level of performance or integration capabilities.
  3. Custom-built solutions for managing vector data, which require significant development effort.

Additional Context

This integration has been well tested with the following code, ensuring reliable performance and accuracy:

import os
import logging
from transformers import AutoTokenizer
from sentence_transformers import SentenceTransformer
from griptape.drivers import QdrantVectorStoreDriver, HuggingFaceHubEmbeddingDriver
from griptape.tokenizers import HuggingFaceTokenizer
from griptape.loaders import PdfLoader
from griptape.chunkers import TextChunker

# Configure logging
logging.basicConfig(level=logging.INFO)

# Setting the models
embedding_model = "sentence-transformers/all-MiniLM-L6-v2"
qdrant_model = SentenceTransformer("all-MiniLM-L6-v2")
file_name = "test_file.pdf"
HUGGINGFACE_TOKEN = os.environ["HUGGINGFACE_HUB_ACCESS_TOKEN"]

# Using HuggingFaceTokenizer
def create_tokenizer(embedding_model):
    tokenizer = HuggingFaceTokenizer(max_output_tokens=1024, tokenizer=AutoTokenizer.from_pretrained(embedding_model))
    return tokenizer

def create_embedding_driver(embedding_model, tokenizer):
    embedding_driver = HuggingFaceHubEmbeddingDriver(
        api_token=HUGGINGFACE_TOKEN, model=embedding_model, tokenizer=tokenizer
    )
    return embedding_driver

# Instantiating QdrantVectorStoreDriver
def create_vector_store_driver(url, collection_name, embedding_driver):
    vector_store_driver = QdrantVectorStoreDriver(
        url=url,
        collection_name=collection_name,
        content_payload_key="content",
        force_recreate=True,
        embedding_driver=embedding_driver,
    )
    return vector_store_driver

# Opening the file
def load_pdf(file_name, tokenizer):
    with open(file_name, "rb") as f:
        loader = PdfLoader(tokenizer=tokenizer, chunker=TextChunker(tokenizer=tokenizer, max_tokens=1024)).load(
            f.read()
        )
    return loader

def main():
    tokenizer = create_tokenizer(embedding_model)
    embedding_driver = create_embedding_driver(embedding_model, tokenizer)
    vector_store_driver = create_vector_store_driver(
        url="http://localhost:6333",
        collection_name="test_collection",
        embedding_driver=embedding_driver,
    )

    # Loading the data
    loader = load_pdf(file_name, tokenizer=tokenizer)

    # Generate metadata for each chunk (example metadata)
    metadata = [{"source": file_name, "page": i + 1} for i in range(len(loader))]

    try:
        for i, l in enumerate(loader):
            content = str(l)
            meta = metadata[i] if metadata else None
            vector_store_driver.upsert_vector(embedding_driver.try_embed_chunk(content), meta=meta, content=content)
        logging.info("Successfully upserted vectors with metadata.")
    except Exception as e:
        logging.error(f"Error during upsert_vector: {e}")

    query_string = "Who created linux?"

    query_results = vector_store_driver.query(query_string, count=6, include_vectors=True)
    # print(f"Query results: {query_results}")
    for result in query_results:
        print(f"ID: {result.id}, Score: {result.score}, Vector: {result.vector}, Metadata: {result.meta}")

if __name__ == "__main__":
    main()

Output

  • Script Output:
Token is valid (permission: write).
Your token has been saved in your configured git credential helpers (osxkeychain,store).
Your token has been saved to /Users/hkhajgiwale/.cache/huggingface/token
Login successful

tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 350/350 [00:00<00:00, 2.20MB/s]
vocab.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 532kB/s]
tokenizer.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 466k/466k [00:00<00:00, 714kB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 112/112 [00:00<00:00, 783kB/s]
WARNING:pypdf._reader:Xref table not zero-indexed. ID numbers for objects will be corrected.
Token indices sequence length is longer than the specified maximum sequence length for this model (566771 > 512). Running this sequence through the model will result in indexing errors

WARNING:pypdf._reader:Xref table not zero-indexed. ID numbers for objects will be corrected.
Token indices sequence length is longer than the specified maximum sequence length for this model (566771 > 512). Running this sequence through the model will result in indexing errors
Inserted a total of 789 vectors into the Qdrant collection 'linux_bible'.
  • Query Result:
ID: 27c79364-047c-4f1a-8323-3da840df95f2, Score: 0.6420436, Vector: [-0.033319965, -0.033909258, -0.014088887, -0.026663534, 0.06426538, -0.08638656, -0.036058135, 0.13097586, 0.06465988, -0.025472835, 0.041268587, 0.01978924, -0.008526307, -0.094854966, 0.047591303, -0.09618401, -0.010990408, -0.027464865, 0.03230282, -0.072785355, -0.039835285, 0.044199828, -0.018777357, -0.010172242, 0.021567283, 0.04779191, 0.030713014, -0.014952045, 0.0342129, -0.021135217, -0.022233197, 0.11858495, 0.026467795, 0.008923172, -0.16770366, 0.061032046, 0.012555608, -0.04789152, -0.08986295, -0.08685627, -0.092642754, -0.056637816, -0.02444664, 0.0148750795, 0.022018181, -0.03189946, 0.00069411576, -0.05575728, 0.007895758, -0.059999734, -0.033308554, -0.040583532, 0.074766226, 0.028454041, 0.02604916, -0.064518526, -0.07666384, -0.04242904, 0.0011933906, 0.034024354, -0.03685598, -0.023715403, -0.06955432, 0.05775433, 0.031021109, -0.052915003, -0.0032551256, -0.036131177, 0.024844615, -0.13610701, -0.040748127, -0.038092654, 0.068615645, 0.10100414, -0.07935713, -0.007989107, -0.055178087, 0.084142484, 0.008011023, -0.0023816512, 0.06601421, 0.14551388, -0.058035214, 0.031445645, -0.048077744, 0.08887344, 0.013803761, 0.085468456, -0.013052811, -0.0697258, 0.048760545, -0.017685851, 0.015883174, -0.037387706, -0.041260507, -0.07488407, 0.089132234, 0.0061633103, 0.031875677, 0.06863733, 0.019712245, 0.0038533423, 0.013222222, 0.015128401, -0.0016742714, -0.026235057, 0.010969684, 0.045371763, 0.026369834, -0.001685722, -0.06804295, -0.04602395, -0.072482534, -0.02093926, 0.007999741, -0.08011011, -0.0267737, 0.020341981, 0.051036708, 0.0038343293, -0.00063315715, 0.026499316, -0.0860942, -0.021107929, 0.06657996, 0.054002028, -0.034046065, 4.5936933e-34, 0.06063078, 0.0041874545, -0.03279631, 0.04840063, 0.012547468, -0.0036221752, -0.007934127, 0.008459876, 0.0031696947, 0.025607022, 0.03910192, 0.040383086, -0.0033453074, 0.039749652, 0.11728535, -0.020587133, -0.059816215, -0.025107108, 0.060377054, 0.012229162, -0.018001657, 0.045607377, 0.031661518, 0.061065547, 0.071817026, 0.012916247, -0.062444888, -0.09693645, -0.016995782, -0.0007742, -0.025836606, 0.026911916, -0.028476752, 0.013248208, -0.0032107763, -0.044497356, -0.008737978, -0.118968375, 0.015046552, -0.04112476, -0.026331374, 0.0098819155, -0.02784592, -0.03217574, 0.08088148, -0.009977335, -0.0031248387, -0.0017853056, 0.02869835, -0.07050618, -0.039792247, 0.0085486155, 0.12939373, 0.014063032, 0.036617287, 0.035789784, -0.036874287, 0.020059178, 0.015771665, 0.16028489, -0.07761481, 0.06122184, -0.021478254, 0.021385562, -0.034210004, 0.036777332, -0.0012745084, 0.01997811, -0.027142646, 0.09602916, -0.06377781, 0.006446841, -0.024087435, 0.07697739, -0.038981833, 0.0531924, -0.014489998, -0.084309846, -0.020455772, 0.039448157, -0.030567326, 0.025330642, -0.019592656, 0.012218679, 0.053152625, 0.055347815, -0.0705193, -0.0059185554, -0.026899993, -0.085054144, 0.058190323, -0.036551163, -0.048517633, 0.07384973, 0.006800745, -4.2462456e-33, 0.020581827, 0.035085097, -0.05775891, -0.018412707, 0.0067332885, 0.05631842, -0.07016557, -0.03622008, -0.057021916, -0.08257919, -0.022888742, 0.01997705, 0.07339474, 0.043937683, 0.0332707, -0.0074235, -0.048726168, -0.053748786, -0.067911476, -0.0339103, -0.055709537, 0.061154872, -0.024023516, -0.08668337, 0.042099033, 0.02462188, -0.0096963635, 0.022291938, -0.04741452, 0.02477046, 0.09467683, -0.05747949, -0.0057552066, -0.0077939606, 0.1068827, 0.012320172, 0.0013870214, 0.028096963, -0.008436847, -0.03088452, 0.03593164, -0.014096904, 0.05320094, -0.0195332, -0.04015384, -0.03202634, -0.09061283, 0.027282033, -0.01103524, -0.10506805, 0.03868709, -0.013999539, 0.002591833, -0.02782352, 0.016672468, -0.10313568, -0.038809996, 0.020939296, 0.0106487945, 0.04167872, 0.012849844, -0.04614982, 0.018726261, 0.08172715, -0.079679236, -0.047526445, -0.059034318, 0.028333595, -0.025118697, -0.018119022, 0.027818607, 0.020889938, -0.027688514, -0.0064164065, 0.021720523, 0.05787196, 0.02540303, -0.051163625, -0.07985068, 0.036325943, 0.03132891, 0.03267965, 0.026869241, 0.04623621, -0.051324327, 0.04002144, 0.007920646, -0.026736474, 0.020703364, -0.02433869, 0.041062724, 0.022844095, -0.044597227, 0.14091592, -0.015348343, -5.287611e-08, 0.06937336, 0.0015003928, 0.0063034287, -0.0015761731, 0.0641838, -0.04084883, 0.069532774, -0.019854434, -0.057561822, 0.04487838, -0.04808787, -0.020828236, -0.039036937, -0.08314201, 0.1117453, 0.08481166, -0.05059345, 0.09510657, -0.03328913, -0.016192343, 0.08916456, -0.13047144, -0.020908123, 0.049791463, -0.01720267, -0.06338032, 0.050439823, 0.012224918, 0.06679014, 0.052619737, -0.028156029, -0.014997442, -0.02212743, 0.036619224, 0.102874845, -0.030618286, -0.030848863, -0.044709735, -0.002172509, -0.039022136, -0.049619682, -0.007745338, 0.0043051774, -0.012630724, -0.074661165, 0.10782143, -0.08718763, -0.01271671, 0.005002096, -0.0023495308, 0.008480507, 0.06494349, 0.021023551, 0.08271026, 0.009580766, -0.029174384, -0.028336622, -0.034248553, 0.023291852, -0.018573066, -0.041182272, -0.01851133, 0.108130485, 0.034235995], Metadata: {'content': 'ated the Linux kernel as a graduate student in Finland) is what gives Linux its name. The kernel is the software that starts up when you boot your computer and interfaces with the programs you use so they can communicate effectively and simply with your computer hardware.\nNote\nSee Appendix B for historic details on how the kernel and other free software came together to create the \nLinux phenomenon. /square6\nComponents such as administrative commands and applications from other free and open source \nsoftware projects work with the kernel to make Linux a complete operating system. The GNU proj-ect (\nwww.gnu.org ), in particular, contributed many implementations of standard UNIX compo-\nnents that are now in Linux. Apache, KDE, GNOME, and other major open source projects in Linux have also contributed to the success of Linux. (See Chapter 2 for an explanation of how open source projects work and how you can get involved in them.) Those other projects added such things as\nGraphical user interfaces (GUIs)\n/circle6 —Consisting of a graphical framework (typically the \nX Window System), window managers, panels, icons, and menus. GUIs enable you to use Linux with a keyboard and mouse combination, instead of just typing commands (as was done in the old days).\nAdministrative utilities\n/circle6 —Including hundreds (perhaps thousands) of commands and \ngraphical windows to do such things as add users, manage disks, monitor the network, \ninstall software, and generally secure and manage your computer.\nApplications/circle6 —Although no Linux distribution includes all of them, literally thousands \nof games, offi  ce productivity tools, Web browsers, chat windows, multimedia players, \nand other applications are available for Linux.Programming tools\n/circle6 —Including programming utilities for creating applications and \nlibraries for implementing specialty interfaces.Server features\n/circle6 —Enabling you to offer services from your Linux computer to another \ncomputer on the network. In other words, while Linux includes Web browsers to view \nWeb pages, it can also be the computer that serves up Web pages to others. Popular server features include Web, mail, database, printer, fi  le, DNS, and DHCP servers.\nAfter Linus Torvalds and friends had a working Linux kernel, pulling together a complete open source operating system was possible because so much of the available “free” software was\nCovered by the GNU Public License (GPL) or similar license\n/circle6 —That allowed the \nentire operating system to be freely distributed, provided guidelines were followed relating to how the source code for that software was made available going forward (see \nhttp://www.gnu.org/licenses/gpl.html ).\nBased on UNIX-like systems/circle6 —Clones of virtually all the other user-level components \nof a UNIX system had been created. Those and other utilities and applications were built to run on UNIX or other UNIX-like systems.\n85057c01.indd   1285057c01.indd   12 10/15/09   2:22:54 PM10/15/09   2:22:54 PM\n13Chapter 1: Starting with Linux\nLinux has become one of the most popular culminations of the open source software movement. \nBut the traditions of sharing code and building communities that made Linux possible started years before Linux was born. You could argue that it began in a comfortable think tank known as Bell Laboratories. Read Appendix B to learn more about the history of Linux.\nLeveraging work done on UNIX and GNU projects helped to get Linux up and running quickly. \nThe culture of sharing in the open source community and adoption of a wide array of tools for communicating on the Internet have helped Linux to move quickly through infancy and adoles-cence to become a mature operating system.\nThe simple commitment to share code is probably the single most powerful contributor to the \ngrowth of the open source software movement in general, and Linux in particular. That com-mitment has also encouraged involvement from the kind of people who are willing to contribute back to that community in all kinds of ways. The willingness of Linus Torvalds to incorporate code from others in the Linux kernel has also been critical to the success of Linux.', 'page': 84, 'source': 'linux_bible.pdf'}
ID: faf99e96-5c1a-47a0-9e5e-cedd04146bbe, Score: 0.5350301, Vector: [0.024331223, -0.07055378, -0.04639513, -0.06745808, 0.11607092, -0.06295789, -0.050465923, 0.13786222, 0.030159013, -0.004015608, -0.010676871, 0.043769825, 0.002562885, -0.10791365, 0.08188917, -0.06733823, 0.015235985, -0.04399143, -0.029254304, -0.07622137, -0.051291894, -0.042619515, -0.035561237, -0.03216422, -0.015353801, 0.007827441, 0.003842683, -0.024111375, 0.011969527, -0.073085405, -0.004015697, 0.05118515, 0.04462064, 0.016457165, -0.044123165, 0.029161211, 0.032725297, -0.09948047, -0.11717596, -0.07006103, -0.07375753, -0.06258994, -0.04566733, 0.07318526, -0.018762583, -0.028633866, 0.00589228, -0.025655907, 0.028555587, -0.05916732, 0.0030022084, -0.028934382, 0.045359496, 0.008084487, -0.0059830844, -0.06705739, -0.039903928, -0.05109666, 0.017328693, 0.019561894, 0.03865932, -0.031135602, -0.094755515, 0.0696307, 0.0079221, -0.0009801845, 0.04914509, -0.044889834, -0.014624922, -0.06107484, -0.08638306, 0.004438856, 0.07317532, 0.059987534, -0.084445775, -0.019608138, 0.03666604, 0.052662563, 0.014499273, 0.026804266, 0.064353116, 0.121469095, -0.095886566, 0.0138858305, -0.061298773, 0.042142395, -0.04137195, 0.016541636, 0.027204953, -0.03736728, 0.024546837, 0.04318625, 0.045745023, 0.018195828, 0.012363759, 0.009674581, 0.07117867, -0.05164778, -0.042451415, 0.10459722, -0.051867876, 0.039014667, 0.053602595, -0.013444189, -0.059383016, 0.026365539, 0.03386706, 0.052461583, -0.05169558, -0.016761262, -0.045771126, -0.05824901, -0.062215872, -0.043441582, -0.0011185831, -0.015508645, -0.07924711, 0.0067165955, 0.027270535, 0.0128110545, 0.055487428, 0.0096251005, -0.052989874, -0.035143983, 0.0745588, 0.008490463, -0.0061789937, 5.7813365e-33, -0.0077322903, 0.05089334, -0.06208537, -0.012271753, 0.037881814, -0.032028474, 0.031108173, 0.03932017, -0.113128036, 0.065454625, 0.04473485, 0.048975635, -0.0397826, -0.041689493, 0.15407364, -0.0093920585, -0.048749406, 0.01576997, 0.039169155, 0.028227244, 0.04160524, -0.009283758, -0.0039213984, 0.014405117, 0.071765065, -0.04113585, -0.06290899, -0.09136639, 0.06928661, 0.020446315, -0.05478173, 0.015546732, -0.06627345, 0.025668463, -0.02507427, -0.054088365, -0.09528659, -0.07856511, 0.02387049, 0.013679139, -0.075428315, 0.030132407, -0.0421513, -0.061015166, 0.039319675, 0.0005365387, 0.010995959, 0.0019172272, -0.0077442573, -0.0067242687, -0.032987166, 0.063831784, 0.07289842, 0.06597161, 0.021873735, -0.015787529, 0.008175407, -0.0051377737, 0.035549764, 0.13740769, -0.07484701, 0.01975612, 0.00097205286, -0.016821997, -0.02086878, -0.043551154, 0.029973619, 0.051265553, -0.009231957, 0.091875225, -0.040937915, -0.020471284, -0.013983976, 0.04625708, -0.011935796, 0.036329255, -0.02307174, -0.050468333, 0.034062546, -0.024977278, -0.053438656, 0.09805345, -0.04715404, 0.06499154, 0.0026042063, 0.048584484, -0.05834566, -0.017755663, -0.054628246, 0.04123106, 0.008707029, 0.0045137308, 0.015643694, 0.07451622, 0.0013959655, -5.6718363e-33, -0.032046035, -0.043305457, -0.0010658664, 0.0074834055, 0.033055622, 0.054477125, -0.08341126, 0.019022267, -0.03950564, 0.022333609, -0.07156558, 0.023374123, 0.03629009, -0.0011669495, 0.016282769, -0.073552124, 0.030347448, -0.055158608, -0.025584424, 0.00715488, -0.058990058, 0.15312871, 0.009879565, -0.097141646, 0.01882934, 0.069251604, -0.08145449, 0.0343667, -0.050875288, 0.019485632, 0.05586577, -0.015215959, -0.016180554, -0.03318689, 0.09547338, 0.023699587, 0.03604217, 0.048568293, -0.012289297, -0.0016615205, 0.08206157, -0.017569851, -0.06479148, -0.052218817, -0.008343782, 0.0065487307, -0.06361596, -0.015336794, 0.010537647, -0.075127974, 0.058192264, 0.011879299, -0.008780829, -0.040178724, -0.0033989523, -0.06746455, -0.044544596, 0.057894103, 0.031869363, 0.030301424, 0.06432371, -0.04518761, -0.047280554, 0.10941228, -0.0032685648, -0.051799525, 0.029878203, 0.086914174, -0.023249878, 0.054577317, -0.048716806, -0.022192033, 0.010497893, 0.007632791, 0.0058831843, 0.07827393, 0.03452274, -0.061196122, -0.061946824, 0.05277168, -0.0036381485, 0.01532919, 0.005320174, 0.056102376, -0.027673684, -0.026463313, 0.045050167, -0.060121257, 0.008630904, -0.05514328, -0.028198322, 0.06288215, -0.06112665, 0.12975036, 0.044362936, -5.1572744e-08, 0.06916056, -0.055261664, -0.0121382205, 0.05772162, 0.013674339, -0.009856258, 0.060743503, -0.0068293344, -0.042229537, 0.12912934, 0.022893893, -0.06768744, -0.034169108, -0.019764515, 0.021298576, 0.15513909, -0.047815453, 0.08062074, -0.06744864, -0.010472879, 0.093221396, 0.0092249, 0.033652484, 0.004951226, -0.0108227655, -0.08298904, 0.07829765, -0.02918012, 0.040861372, 0.07502183, -0.013328497, -0.007117863, -0.0010515214, 0.016050711, 0.07137554, -0.05032972, -0.055674076, 0.05260627, 0.06055885, 0.010967028, -0.044464994, 0.08605771, -0.008431955, -0.007856606, -0.08918092, -0.03736765, -0.10412424, -0.006117804, -0.0065842066, -0.035468876, -0.008603244, 0.017469272, -0.022496583, 0.070861705, 0.059056558, -0.026201164, -0.015157915, 0.02911126, 0.04120862, 0.038511287, -0.037579004, -0.0666424, 0.042541374, 0.0468446], Metadata: {'content': 'www.linuxdevices.com ) to game consoles (such as PlayStation 3) to \nsupercomputers to Mars rovers. In short, Linux has become a system that almost anyone can run almost anywhere.\nOn both desktop and server computers Linux has become a formidable \noperating system across a variety of business applications. Today, large enterprises can deploy thousands of systems using Linux distributions from companies such as Red Hat, Inc. and Novell, Inc. Small businesses can put together the mixture of offi  ce and Internet services they need to keep their \ncosts down.\nThe free and open source software (FOSS) development model that \nespoused sharing, freedom, and openness is now on a trajectory to sur-pass the quality of other operating systems outside of the traditional Linux servers and technical workstations. What were once weak components of Linux, such as easy-to-use desktops, multimedia codecs and limited driver availability, have improved at a rapid pace. In areas of security, usability, connectivity, and network services, Linux has continued to improve and outshine the competition.Starting with Linux\nIN THIS CHAPTER\nGetting started with Linux\nUnderstanding LinuxLinux features and advantages\n85057c01.indd   385057c01.indd   3 10/15/09   2:22:53 PM10/15/09   2:22:53 PM\n4Part I: Getting off the Ground with Linux\nComputer industry heavy-hitters such as Microsoft and Oracle have taken notice of Linux. \nMicrosoft has struck agreements with Linux companies including Novell and Xandros to form partnerships that primarily protect those companies against threatened Microsoft lawsuits. Oracle began producing its own enterprise-targeted Linux system to try to stem the fl  ow of cus-\ntomers to Red Hat Enterprise Linux.\nWhat does this all add up to? A growing swirl of excitement around the operating system that the \nbig guys can’t seem to get rid of. For people like yourself, who want the freedom to use your com-puter software as you like, it means great prospects for the future.\nLet this book help you grab your fi  rst look at the distributions, applications, services, and com-\nmunity that make up the phenomenon that has become Linux.\nTaking Your First Step\nIn your hands, you have 18 different Linux distributions (on CD and DVD), thousands of applica-\ntions, and descriptions for getting it all running on your own computer. For you right now, the worldwide Linux phenomenon is just a reboot away.\nLinux Bible 2010 Edition brings you into the world of free and open source software that, through \nsome strange twists and turns, has fallen most publicly under the “Linux” banner. Through descriptions and procedures, this book helps you\nUnderstand what people do with Linux and how you can use Linux for all your comput-', 'page': 76, 'source': 'linux_bible.pdf'}
ID: 559b54c1-b769-4c3c-b2cd-f832a93fb99f, Score: 0.52532214, Vector: [0.00746728, -0.11844351, -0.007997972, 0.011819327, 0.16083239, -0.027890231, -0.030016694, 0.16784942, 0.009562944, 0.008335069, -0.013510524, 0.011549256, -0.002929275, -0.11596569, 0.076392695, -0.05113839, -0.09048634, -0.054655273, 0.030377481, -0.093296915, -0.003533422, 0.01996293, 0.03350387, -0.027454238, 0.030402206, 0.028024137, 0.027541379, -0.07663866, 0.034651108, -0.044327524, 0.0043478194, 0.1217768, 0.015323285, -0.011632312, -0.06851035, 0.025471015, 0.07107715, -0.06755785, -0.054285582, -0.020122241, -0.1076624, -0.043894358, -0.039558616, 0.012672985, 0.014954614, -0.041535545, -0.0032710384, -0.042225473, 0.013539396, -0.052253213, -0.03387083, -0.05464647, 0.0065770554, 0.019909745, 0.012807563, -0.04035854, -0.01783124, -0.09215303, -0.015678443, -0.07305397, 0.04806908, -0.051449973, -0.10381545, 0.069721416, -0.04161106, -0.014806088, -0.007637158, 0.077433415, 0.055192057, -0.090190195, -0.07149286, -0.03136405, 0.058312405, 0.08901262, -0.0028556655, -0.02804934, -0.006143313, 0.05121465, 0.019295719, 0.055265322, 0.05419187, 0.069835, -0.052713167, 0.05128869, -0.08664806, 0.059907466, 0.022448724, 0.038762752, 0.0010765686, -0.049246322, 0.039219253, 0.050062515, 0.013636117, -0.015030285, 0.019150577, -0.034392092, 0.07742463, -0.0508219, -0.0017189122, 0.09155765, -0.04176071, -0.008775968, -0.009357665, -0.026293984, -0.07783781, 0.015077545, 0.02026658, 0.025035856, 0.0037758523, -0.013756253, -0.017237144, -0.018613083, -0.028278133, -0.044136588, 0.021040194, -0.028701229, -0.050071683, -2.1271697e-05, -0.0068169152, 0.050660476, -0.020379832, 0.06683549, -0.04830146, -0.0068403357, 0.06609714, 0.03009027, -0.040599417, 1.713397e-33, 0.08282234, 0.085729845, -0.049530335, -0.0024655517, 0.02111573, -0.05216762, -0.0014215877, 0.03463404, -0.0848725, 0.08374194, 0.10646674, 0.101235144, -0.067771494, -0.017541744, 0.08444848, -0.07616206, -0.058477767, -0.022663856, 0.03657865, 0.0052248565, -0.009666332, 0.019909766, -0.00357673, 0.042327613, 0.013103364, -0.07006073, -0.05037492, -0.08474202, 0.045402613, -0.00415185, -0.08691769, 0.053092122, -0.052869864, -0.04253742, -0.041239712, -0.044962313, -0.07403936, -0.08684757, 0.042667765, -0.016743267, -0.06625975, 0.03955627, 0.011500935, -0.05137156, 0.062236868, 0.008678709, 0.071807645, 0.01332741, 0.054589562, -0.0337019, -0.07072625, -0.033138905, 0.016805813, 0.07812944, -0.03546032, 0.029750424, -0.020088248, -0.03131388, 0.02462189, 0.10450523, -0.019727007, 0.010109403, 0.027690204, -0.047882847, -0.06731156, 0.007277827, 0.044157535, 0.036198664, 0.012190073, 0.07966004, -0.058520474, -0.04031732, -0.02942285, 0.052512333, -0.041164894, 0.05458101, -0.01795128, -0.04443241, 0.01644369, -0.00730637, -0.054506686, 0.06160984, -0.02327273, 0.032622233, 0.05859294, 0.04919139, -0.0384079, -0.072201855, -0.063880034, 0.032740485, 0.08284124, -0.034969304, -0.043666027, 0.054361627, 0.0041896417, -3.5277522e-33, 0.020163145, -0.059797283, 0.0012108752, -0.0014657109, 0.04875801, 0.060138997, -0.09107212, 0.038635015, 0.013244924, -0.017910503, -0.0833695, 0.011083091, 0.04996642, 0.027823742, 0.0013634168, -0.094470024, 0.021212595, -0.015475093, -0.07432784, 0.00825962, -0.04147317, 0.08459714, -0.020067075, -0.11088987, 0.07192243, 0.045834888, -0.021038346, 0.027490195, -0.046723317, 0.08015263, 0.09808546, -0.029732263, -0.022613453, -0.0022434823, 0.052874245, 0.0043748957, 0.049254607, 0.025183193, 0.0063277683, -0.013529587, 0.06683558, 0.016061414, -0.0039460785, -0.080721654, -0.018246287, -0.024714625, -0.09032139, 0.019064434, 0.008227884, -0.104419395, 0.028195327, -0.012441221, 0.06800411, -0.06576471, 0.015530077, -0.04920336, -0.039293244, 0.043917872, 0.08952696, 0.010601837, -0.018011035, -0.0034346264, -0.046036497, 0.0957672, -0.042801708, -0.10759926, 0.009244587, 0.01889225, -0.013532307, -0.005812693, -0.054307435, 0.033654302, -0.023743318, -0.030277519, -0.0240804, 0.06364629, 0.00070645014, -0.025784686, -0.09093685, -0.010048218, 0.03863883, 0.037403032, 0.0033992024, 0.014415501, -0.047117464, 0.04075601, 0.08425122, -0.041142225, -0.007136692, -0.02124992, -0.025833722, 0.072112374, -0.0085339425, 0.099372335, 0.03907196, -5.1821072e-08, 0.043728434, -0.050226863, -0.019331686, 0.08265944, 0.0062530763, 0.00013409385, 0.032990552, -0.00386631, -0.029465502, 0.10428476, -0.022173971, -0.03215279, -0.018030467, 0.0009613294, 0.052704975, 0.09792233, -0.029748386, 0.059049904, -0.058105685, -0.032421697, 0.13109277, -0.02589744, 0.06618135, 0.021598492, 0.009302079, -0.061946012, 0.07017781, -0.0001900025, 0.053011436, 0.03306959, -0.0047116936, 0.019234378, -0.05053415, -0.017033761, 0.08826063, -0.0536139, -0.027290963, 0.03367373, 0.049046386, 0.026319882, -0.074161656, 0.0363738, 0.014411307, -0.019714253, -0.100796, 0.01724432, -0.06411304, 0.020741925, 0.03345999, -0.006161178, 0.04991325, -0.013879994, 0.03182722, 0.02034885, 0.03105859, 0.041133456, -0.06589748, 0.08011253, 0.04666577, -0.0033899012, -0.04469346, -0.079689756, 0.011030314, 0.042117614], Metadata: {'content': 'In the fi  rst two chapters, you’ll learn a lot of exciting ways Linux is being used today and see who \nmany of the major players are in the free and open source software (FOSS) world. You will see \nhow people are adapting Linux to run on handhelds, mini laptops, 32- and 64-bit PCs, Macs, mainframes, and super computers. Linux is truly everywhere!\nHowever, if you are concerned that somehow “free” software is too good to be true, skip ahead for \nthe moment to the brief history of Linux in Appendix B. That appendix guides you through the strange and circuitous path of free and open source software development that led to the Linux phenomenon.\nIf you are intrigued by what you learn here, I’ll tell you how you can become part of the open \nsource and free software communities, whose stars are known by a single name (such as Linus) or a few initials (such as rms). You’ll fi  nd a staggering number of open source projects, forums, \nand mailing lists that are thriving today (and always looking for more people to get involved). \nHow This Book Is Organized\nLearn the basics of what goes into Linux and you will be able to use all sorts of devices and computers in the future. The book is organized in a way that enables you to start off at the very beginning with Linux, but still grow to the point where you can get going with some powerful server and programming features, if you care to.\nPart I includes two short chapters designed to open your eyes to what you can do with Linux, \nthen get your hands on it quickly. Those two chapters describe\nHow others use Linux, how to transition to Linux from Windows, and how to start with \n/circle6\nLinux using the CD and DVD inside this book (Chapter 1)\nWhat you can do, what you can make, and what you can become with Linux (Chapter 2)/circle6\n85057flast.indd   xxvi85057flast.indd   xxvi 10/15/09   2:23:00 PM10/15/09   2:23:00 PM\nxxviiIntroduction\nIn Part II , you start in with details on how to use Linux desktops and associated applications. \nChapters 3–7 describe\nThe KDE, GNOME, and other desktop interfaces (Chapter 3)/circle6\nTools for playing music and video (Chapter 4)/circle6\nDesktop publishing and Web publishing using word processing, layout, drawing, and /circle6\nimage manipulation tools, plus tools such as wikis, blogs, and content management sys-\ntems for managing content online (Chapter 5)\nApplications for e-mail and Web browsing/circle6  (Chapter 6)', 'page': 73, 'source': 'linux_bible.pdf'}
ID: edbaf079-9d34-4de7-b38b-e79ec9a16962, Score: 0.48150444, Vector: [-0.024073696, -0.06740264, -0.04933037, -0.056596126, 0.0640839, -0.08332419, 0.0027693151, 0.13868594, 0.057942413, 0.0045598065, 0.0018336017, 0.07531288, 0.018394172, -0.07559044, 0.0490967, -0.073607825, -0.013805403, -0.053357575, 0.028111273, -0.089254394, -0.05395274, 0.016952382, -0.024828944, 0.0049645705, 0.016095687, -0.0074422187, -0.0111827785, -0.0129252635, 0.015384097, -0.02754346, 0.027828624, 0.051317822, -0.024497803, -0.017709127, -0.070052184, 0.070469335, 0.043555737, -0.04935626, -0.043799397, -0.050337467, -0.070827074, -0.024127513, -0.020474128, 0.038182694, 0.0008033708, -0.03626968, -0.050244138, -0.014230472, -0.027392898, -0.05233938, -0.05788181, -0.0087971985, 0.07703545, 0.03716889, 0.0056894175, -0.08257068, -0.006536877, -0.048928753, -0.0030512463, -0.042867243, 0.026243336, -0.035939734, -0.084637694, 0.065710865, 0.001997814, -0.0055882614, 0.006594209, 0.02063613, 0.016229676, -0.09070771, -0.09010035, -0.047365617, 0.036659993, 0.11633981, -0.039124742, -0.021918172, 0.027612792, 0.007670698, 0.01996161, 0.014145673, 0.032564282, 0.120462574, -0.002979826, 0.050316617, -0.077059455, 0.06513185, 0.06011874, 0.05446738, 0.05895109, -0.047661755, 0.09689729, 0.04157286, 0.0069837584, -0.025893275, -0.04024964, -0.00027525637, 0.08663912, -0.063367985, -0.016695589, 0.07634073, -0.0046042968, -0.07553812, 0.016132131, -0.06466097, -0.046718005, -0.012522675, 0.040737897, 0.029054755, 0.0054848413, 0.020392884, -0.12974729, -0.04055431, -0.09623701, -0.024364535, -0.020406479, -0.076892, -0.06804503, 0.040718183, 0.031146491, 0.0005500672, 0.02577542, 0.059472144, -0.037667777, -0.06337242, 0.09070736, 0.011583493, 0.024941992, 2.911553e-33, 0.015209375, 0.05031743, -0.071215, -0.043480393, 0.027895976, -0.065813914, 0.0075986134, 0.045926567, -0.078338884, 0.05116739, 0.023592697, 0.06374172, -0.026450198, 0.016269743, 0.17122796, -0.021455862, -0.03337701, -0.01743979, 0.0930162, -0.03729465, -0.04721721, 0.010541696, 0.037904505, 0.023485055, 0.03921805, -0.05274242, -0.005406335, -0.034132645, 0.015480098, -0.01049564, -0.016216632, 0.011952389, 0.0054787714, -0.012547889, 0.0142544545, -0.01064859, -0.087488934, -0.09501916, 0.027134314, 0.007356856, -0.029017627, 0.036929734, -0.0067986315, 0.024368, 0.07348975, 0.03733985, 0.028452594, -0.021815332, 0.014123607, 0.013288786, -0.077107206, -0.020698708, 0.061511688, 0.042465787, -0.07053348, 0.053588647, 0.022194672, -0.033159703, 0.02148347, 0.083170116, -0.061057176, -0.051194493, 0.03694642, -0.05161562, -0.027916215, -0.060474515, 0.05438269, -0.002920457, 0.036952972, 0.05566183, -0.060088944, -0.06387428, 0.012257671, 0.03642768, -0.023741508, -0.0056332806, 0.009154998, -0.052247345, 0.023426965, 0.019512601, -0.09632673, 0.032396507, -0.06482864, 0.040396318, 0.08001134, 0.060092065, -0.09680037, 0.03586784, -0.021762831, 0.006366735, 0.038419466, -0.04474926, -0.014793833, 0.10049441, 0.0023396981, -4.9488852e-33, -0.08328449, -0.07763805, 0.0026429582, 0.04993892, 0.04945391, 0.050035976, -0.10315616, -6.121183e-05, -0.026305752, -0.008961132, -0.059943065, -0.0110602295, -0.002902582, 0.007216774, -0.0017162564, -0.07697368, -0.014351343, -0.029406307, -0.0030815566, 0.016400246, -0.04912495, 0.09657606, -0.004924246, -0.050857008, 0.032611314, 0.016854322, -0.028242879, 0.043756545, -0.0022887944, 0.016687715, 0.11364286, -0.047498297, -0.03072822, -0.011637853, 0.03166923, -0.015000611, 0.104585, 0.012948129, -0.035104573, -0.0007573287, 0.07480522, 0.017861715, -0.04618072, -0.06523287, -0.033361565, -0.02468294, -0.049310334, -0.012103423, -0.009812822, -0.0723652, 0.0133117335, -0.052323323, 0.059941348, -0.013579973, 0.0046214564, -0.041455805, -0.027436167, 0.05619888, 0.06613538, 0.03857493, 0.036038622, -0.023564767, 0.006929671, 0.12132061, -0.06090688, 0.0015805558, 0.05244593, 0.0139290495, -0.078118846, 0.04538711, -0.030370511, 0.03300274, -0.0024404058, -0.050177578, -0.04499654, 0.041831892, -0.026301343, -0.029057309, -0.101833805, -0.010659004, 0.008464277, 0.0019811208, -0.021909859, 0.03556323, -0.07210136, -0.05506824, 0.07401727, -0.09481533, 0.0062993583, -0.06668644, -0.011408364, 0.041455653, -0.031962853, 0.13377553, 0.028018326, -5.7048968e-08, 0.07315667, -0.099995196, 0.0065096393, 0.016581548, 0.012141264, -0.01728705, 0.03933864, -0.00065708964, -0.05199344, 0.074340224, 0.0500053, -0.07451712, -0.07489221, -0.042417005, 0.06900265, 0.11055208, 0.022807684, 0.12333708, -0.05916859, 0.0011295219, 0.102453664, -0.06302064, 0.08926409, 0.0017882853, 0.06728807, -0.04813749, 0.090594076, 0.026217436, 0.029793752, 0.0327958, -0.002881321, -0.026205432, 0.027978938, 0.046610203, 0.09486055, -0.02050504, -0.08508389, -0.03376538, 0.039941408, 0.07671597, -0.03209068, 0.039046388, 0.018367276, 0.006731865, -0.04051718, 0.0066023828, -0.119661875, -0.016749056, 0.042518236, -0.034705643, 0.051088512, -0.033405125, 6.3934596e-05, 0.038810138, 0.06264515, 0.007958509, 0.021373093, 0.01930552, 0.04081616, 0.093892, -0.024699429, 0.0017280468, 0.084522046, -0.030239053], Metadata: {'content': 'confi gure system services, and secure your computers and networks.\nA Linux server administrator/circle6 —Using some of the world’s best server software, you \ncan set up your computer to be a Web server, fi  le server, mail server, or print server.\nA software developer/circle6 —You can draw on thousands of open source programming tools \nto develop your own software applications.\nThe Linux systems you have in your hand don’t contain trialware or otherwise-hobbled software. \nOn the contrary, they feature software created by world-class development projects, the same teams that build the software that powers many professional businesses, schools, home desktops, and Internet service providers. In other words, this truly fi  rst-rate software is from develop-\ners who have made a commitment to producing software that can be used in the ways that you choose to use it.\nMost of the Linux distributions offered on the DVD and CD that come with this book are live \nCDs that let you try a Linux distribution without installing. Almost all of those live CDs include features that let you install the contents of those live CDs to your hard disk. For example, you can try out Fedora, Gentoo, Ubuntu, openSUSE, PCLinuxOS, and Mandriva as live CDs, and then install those distributions permanently to your hard drive from icons on the desktops of those live CDs.\n85057flast.indd   xxv85057flast.indd   xxv 10/15/09   2:22:59 PM10/15/09   2:22:59 PM\nxxviIntroduction\nUnlike some other books on Linux, this book doesn’t tie you to one Linux distribution. The book \nteaches you the essentials of Linux graphical desktop interfaces, shell commands, and basic sys-tem administration. Separate chapters break down many of the major Linux distributions avail-able today. Then descriptions of the major software projects in most Linux distributions (KDE and GNOME desktops, Apache Web servers, Samba fi  le and printer sharing, and so on) guide \nyou in setting up and using those features, regardless of which Linux you choose.\nUnderstanding the Linux Mystique\nThis book is designed to spark your imagination about what is possible with Linux, then give you the software and instruction to jump right into Linux. From there, the approach is to help you learn by using it.', 'page': 72, 'source': 'linux_bible.pdf'}
ID: c83dcac1-b65a-44cc-a261-cae57bfc95d9, Score: 0.47938925, Vector: [-0.01984242, -0.009251191, 0.018457491, -0.03781013, 0.09261274, -0.1193788, -0.07727213, 0.12620927, 0.061431658, -0.008344117, 0.048700366, 0.061782468, 0.020591466, -0.08948288, 0.044110827, -0.04765848, 0.030769775, -0.042209804, -0.009124599, -0.0050316285, -0.027673768, 0.03784497, 0.01110022, -0.028833224, -0.03238227, -0.013826247, -0.013972005, -0.023391213, 0.045812063, -0.024594031, -0.031893104, 0.100631386, 0.05064631, 0.027371014, -0.046596188, 0.016476015, -0.012597444, -0.07095902, -0.12446081, -0.0748582, -0.03965642, -0.09973392, 0.000987572, 0.055254634, -0.010403949, 0.008403102, 0.027986538, -0.08947484, 0.022214811, -0.048560645, 0.007943433, 0.024376217, 0.041964628, -0.0047879405, -0.107106864, -0.08339744, -0.08268022, 0.0054734685, 0.06822181, 0.014187082, 0.068419866, 0.0043118214, -0.06377442, 0.039710492, -0.022174219, -0.018763384, -0.028919643, -0.08097144, 0.019838884, -0.11666613, -0.001925492, -0.054298107, 0.05674856, 0.064811416, -0.1130536, 0.055029638, 0.062297072, 0.03982368, 0.009177934, 0.0065956744, 0.07105321, 0.046975676, -0.05081734, -0.019804347, 0.025514895, 0.04018937, -0.03603348, 0.09892643, -0.015842075, -0.07272974, 0.028053425, 0.022715459, -0.024910647, -0.014285808, -0.062179208, -0.019571653, 0.050337255, -0.048734803, -0.044195447, 0.061492193, -0.034729943, 0.03594042, 0.004427827, 0.046833638, -0.0811618, -0.065017976, -0.0058287294, 0.038173143, 0.01884109, -0.0010024076, -0.03490026, -0.09206027, -0.042224646, -0.023313448, -0.008650974, -0.068659246, -0.07769698, 0.024585923, 0.07928828, 0.0507452, -0.02071425, -0.007857347, -0.054172, -0.085780405, 0.09351972, 0.026023535, -0.021894667, 2.9005352e-33, 0.009481408, 0.039077844, -0.058403946, -0.03236557, -0.0436744, -0.044231296, 0.013088542, -0.0015156794, -0.007417522, 0.07432986, 0.019383166, 0.05898135, -0.048191715, 0.05201061, 0.13181533, 0.01023683, -0.03131462, -0.094403155, 0.033403378, 0.02029259, -0.005610449, -0.020971322, 0.0025865599, 0.058985524, 0.10762909, -0.02880986, -0.009944965, -0.08452161, 0.042229496, 0.013476404, -0.11522525, -0.0024188985, -0.043511253, 0.0072523504, -0.020189347, 0.008922283, -0.097898155, -0.07456626, 0.04497631, -0.013212123, -0.054286465, 0.018808616, -0.021918701, -0.037321456, 0.009258915, -0.009115823, 0.026472611, 0.0015630679, -0.008941925, -0.0698581, -0.023157999, 0.04200182, 0.059763074, 0.031845745, 0.028716076, -0.06301514, 0.0072998507, -0.04026639, 0.035564926, 0.07956249, -0.034908388, 0.025232004, -0.013495203, 0.053126812, -0.049734563, -0.013398143, 0.060294926, -0.0136634465, -0.06469721, 0.111679986, -0.013956184, -0.020290876, -0.037272546, -0.010292192, -0.015074762, 0.057141107, 0.016747905, -0.04238848, 0.015225668, 0.0037201268, -0.010678778, 0.025131961, 0.0011383804, 0.076179445, 0.0059719067, 0.010766955, -0.08781016, -0.052099165, -0.01751505, 0.03523437, -0.014993603, -0.01481304, -0.05686633, 0.09258094, -0.014618604, -3.5376932e-33, -0.06829073, -0.05625367, 0.01898682, -0.018509958, 0.0141084455, 0.033197742, -0.006314207, -0.06868979, -0.057048954, -0.018725408, -0.09629635, 0.020050153, 0.06967098, 0.014687666, 0.040810168, -0.04380362, -0.020860473, -0.05163723, 0.035567094, -0.0116340425, -0.037055183, 0.1415086, 0.02216848, -0.032433484, 0.046441674, 0.0015570261, -0.03556, 0.033511095, -0.030493163, -0.034631, 0.0648899, -0.060514897, 0.024986625, 0.013376488, 0.13324007, 0.058048494, 0.056575127, 0.0039180466, 0.0019940243, -0.049429327, 0.112915434, -0.0021007357, -0.005317771, -0.0010912868, 0.024768947, -0.04099412, -0.10867125, 0.01139904, 0.010392324, -0.043638453, 0.06804612, 0.023437329, -0.0022319213, -0.027865846, -0.031080289, -0.051383488, 0.004178208, 0.016367517, 0.0064936453, -0.005538669, 0.10007223, -0.0513617, -0.014779815, 0.00809466, -0.092387974, -0.035935618, 0.07260038, 0.04789249, -0.030158034, 0.042474665, -0.0046493723, -0.024117976, 0.0284292, 0.015506525, 0.012158536, 0.11800204, -0.0076966677, -0.084179156, -0.02120884, -0.024012763, -0.0050964435, 0.036493853, 0.08855071, 0.02996459, -0.01967626, 0.029032387, 0.0044003217, 0.013221407, -0.0519412, -0.024593018, 0.017867127, -0.022305354, -0.07073401, 0.14142574, -0.050426587, -5.4846257e-08, 0.12618408, -0.027908817, 0.041142188, -0.011754587, -0.044638153, -0.039858174, 0.10112651, -0.07076211, 0.049010936, 0.09001549, 0.055804297, -0.0926332, -0.039273467, 0.03511042, 0.058552198, 0.17512369, 0.012697251, 0.05950258, -0.028028043, 0.022259632, 0.08422044, 0.008618497, 0.008055855, 0.020150276, 0.03209303, -0.029780453, 0.022748128, -0.037858676, 0.040491004, 0.005359437, -0.03923957, -0.02287067, 0.057957288, 0.05262032, 0.051870637, -0.04273112, -0.024588205, -0.0257344, 0.0607741, -0.0013320881, -0.002911388, 0.013258696, -0.038104214, 0.0073573994, -0.09144806, -0.047250293, -0.08016776, -0.08225485, -0.0037114276, 0.07447247, 0.012219077, -0.00036026005, 0.014231857, 0.024624372, 0.06254016, -0.0037182071, 0.025127977, -0.019237185, -0.009543065, 0.013651966, -0.053267457, -0.03690494, 0.06738827, 0.046157088], Metadata: {'content': 'lowing the progress of the Mars Exploration Rover Mission, refer to the project’s Web site at the NASA Jet Propulsion Laboratory (\nhttp://marsrovers.jpl.nasa.gov ).\nLinux in gadgets\nLots of commercial communications, entertainment, and other kinds of gadgets have Linux run-ning on the inside. Linux makes an excellent operating system for these specialty devices, not only because of its cost, but also because of its stability and adaptability. \nLinux enthusiasts love these devices, referred to as embedded Linux systems, because they can \noften adapt, add, or run different Linux software on these devices (whether the manufacturer \nintended that or not). More and more, however, manufacturers are embracing the Linux enthusi-ast and hardware hacker and selling open devices for them to use. Here are some examples:\nMobile Phones\n/circle6 —OpenMoko ( www.openmoko.com ) produces mass-market mobile \nphones, such as the Neo FreeRunner, that are based on Linux. Like the software, the FreeRunner’s hardware also follows an open design. Although the phone is intended for general consumer use, the phone’s software is currently most appropriate for people who want to develop their own software for the phones. Figure 2-2 shows an example of the Neo FreeRunner.\n85057c02.indd   1785057c02.indd   17 10/18/09   11:26:05 PM10/18/09   11:26:05 PM\n18Part I: Getting Off the Ground with Linux\nFIGURE 2-2\nModify software to use the Linux Neo FreeRunner as a phone, GPS device, clock, game player, and media \nplayer.\nMotorola, OpenMoko, and Tranzda Technologies each offer multiple Linux-based \nmobile phones. Phone models running Linux on the inside include the Motorola Rokr EM30 (emphasizing music playing), Tranzda Technologies NewPlus phones (with WiFi, GPS, and a camera), Purple Labs Purple Magic phones (sub-$100 phone), and Grunig B700 (with keyboard and e-mail support).\nSony PlayStation\n/circle6 —Not only can you install and run Linux on PlayStation, but Sony \nencourages you to do it. In 2002, Sony released Linux Kit for PlayStation 2. Included \nin that kit is a derivative of the Japanese Kondara MNU/Linux (which is based on Red Hat Linux). For PlayStation 3, several Linux distributions have been modifi  ed (ported) \nto run on that hardware, including Fedora, OpenSUSE, Ubuntu, Gentoo, Debian, and a commercial Yellow Dog Linux product for PlayStation 3. (See Chapter 22 for a descrip-tion of Yellow Dog Linux.) \nPersonal Video Recorders (PVR)\n/circle6 —If you have a TiVo PVR or a set-top for streaming \nvideo from Netfl  ix, you are already running Linux in your home. The Netfl  ix PVR is \nfrom Roku, Inc. ( www.roku.com ), which produces a range of Linux-based media play-\ners. TiVo has produced Linux-based PVRs for years. The availability of the TiVo Linux \nsource code ( www.tivo.com/linux ) has made TiVo one of the most popular devices \nfor Linux enthusiasts to hack.\nNetbooks/circle6 —Shrinking laptops with shrinking prices have led to Netbooks. These \nmini laptop computers have proven to be excellent devices for running Linux. With \nlow-powered processors and small screens, Netbooks provide a good partnership with Linux systems that are tuned for these compact, effi  cient devices. \n85057c02.indd   1885057c02.indd   18 10/18/09   11:26:05 PM10/18/09   11:26:05 PM\n19Chapter 2: Linux Projects, Activities, and Careers\nThe Asus Eee PC is one of the most popular Netbooks available today. Many Asus Eee \nPCs have been sold with Xandros Linux preinstalled. However, industrious Linux enthusiasts have created ports of Fedora, Ubuntu, and other Linux distributions to run on the Eee PC. The Acer Aspire One is another popular netbook. Figure 2-3 shows an Eee PC with Fedora Linux running on it.\nFIGURE 2-3\nAn Asus Eee PC can run specially tuned Fedora or other Linux systems.\nPersonal handheld devices/circle6 —A whole range of personal digital assistants (PDAs), por-', 'page': 88, 'source': 'linux_bible.pdf'}
ID: e94f41b3-aebb-44a7-8628-2116e344541e, Score: 0.47909445, Vector: [-0.010300992, -0.06201704, -0.046429355, -0.020711467, 0.15296702, -0.06355586, -0.064711764, 0.12849095, 0.0124012, -0.007829238, -0.015711484, 0.08028847, -0.027482616, -0.074537545, 0.06093343, -0.025529772, -0.030304754, 0.006690845, -0.0096968, -0.060203716, -0.03571217, 0.08405562, 0.038551856, -0.017983817, -0.004565045, -0.04740941, -0.03339397, -0.053387243, 0.032289147, -0.06196493, -0.047502507, 0.10740722, -0.011146251, 0.0006463854, -0.001375539, 0.041849215, 0.07455948, -0.047944102, -0.05673705, 0.019862773, 0.0066904738, -0.05481761, 0.0051639997, 0.02902823, -0.022243315, 0.011124962, -0.037202362, 0.0013854739, -0.032971192, -0.0039838725, -0.06632171, -0.044125546, 0.04338726, -0.011582755, 0.057951145, -0.12746732, -0.03482658, -0.05431481, 0.044428058, -0.0540241, 0.075629696, 0.050348457, -0.10716757, 0.055063426, -0.044008967, -0.018426618, 0.0520913, 0.030205825, -0.0060597276, -0.12652613, -0.040145557, -0.01656209, 0.0632583, 0.09220716, -0.08039693, 0.030515926, 0.04136586, 0.060651947, 0.029749708, -0.07679328, 0.04748961, 0.09376876, -0.013562662, 0.019536197, -0.06977183, 0.021640256, 0.038197275, 0.09130066, 0.027071746, -0.0029820877, 0.045834366, 0.026631517, 0.07354314, -0.058127504, 0.023106199, 0.010141544, 0.010416328, -0.063011095, 0.0074689905, 0.08638861, -0.0062448145, -0.00063023536, -0.054977197, -0.049482856, 0.020968132, 0.036079757, -0.023055127, 0.08435904, -0.0077255587, 0.03469405, -0.008138906, 0.010986437, 0.006401985, -0.07772589, 0.07231693, -0.023943482, -0.029293371, -6.1269902e-06, 0.043515585, 0.025842533, 0.009960307, 0.0696169, -0.08994871, -0.051646095, 0.019839454, -0.0019147018, -0.019577768, 2.4558647e-33, 0.051334754, 0.121646635, -0.05315276, -0.010768901, 0.04252973, -0.052605424, -0.020486953, 0.0245449, -0.07791334, 0.03434429, 0.02008067, 0.07927704, -0.04048923, -0.052928265, 0.016566465, -0.03802563, -0.055800743, 0.028523887, -0.016138408, 0.018254908, 0.022699669, -0.010526441, -0.017334113, 0.07162002, 0.0062123733, -0.010634764, -0.037513033, -0.10960199, 0.027357003, 0.0015277078, -0.050305974, 0.00623781, 0.0154764345, -0.011034626, 0.006919724, 0.014764883, -0.111299485, -0.07058621, 0.013226075, -0.0148951905, -0.05644188, -0.008299337, -0.054403916, 0.012482914, 0.10260806, 0.010018166, -0.03929273, -0.12071573, 0.019646032, -0.13366917, -0.006228709, -0.009613113, 0.07321811, 0.026708957, 0.015480196, 0.04757488, 0.0016601183, -0.078985006, 0.041134793, 0.081338294, 0.022034533, 0.08186449, 0.0029554574, -0.07316424, -0.10174299, -0.010351725, -0.041926827, 0.019396342, -0.036740974, 0.059312947, 0.0062836953, -0.03918396, -0.037556905, 0.068071246, -0.12185273, 0.052621372, -0.023773942, -0.080995604, -0.0044211005, -0.017488264, 0.0210193, 0.027639898, 0.013936339, -0.024074338, 0.015879748, 0.003967011, -0.061624024, -0.02687933, -0.034902807, 0.06535775, 0.05245469, 0.021223512, -0.03312717, 0.04180811, 0.009369041, -3.6707265e-33, -0.07170997, -0.049262974, 0.005436494, 0.02097835, -0.017825106, 0.051876474, -0.080931135, 0.023526987, 0.030140061, -0.0046952427, -0.047185495, -0.02370666, 0.02544702, -0.0033252586, 0.010368276, -0.067364074, 0.029264536, -0.06660196, -0.06475247, 0.08037294, -0.051682677, 0.02745868, -0.09859848, -0.044254925, 0.104418464, 0.050720062, 0.033178568, -0.0054844175, -0.034017157, -0.006020143, 0.04121809, -0.045161247, -0.06913875, 0.011488584, 0.061367203, 0.015433793, 0.050012846, 0.00257793, -0.01239022, -0.04589155, 0.07498995, -0.0022535587, 0.020352928, 0.017522484, -0.03924848, -0.03774075, -0.07311946, 0.032017548, -0.012868123, -0.06858843, 0.021572534, -0.02578503, 0.09557191, -0.036143307, -0.026867026, 0.0065293913, -0.04974923, 0.017149953, 0.032028034, 0.06404519, 0.013959068, 0.043303844, -0.11753003, 0.08651047, -0.019341186, -0.10872248, -0.0069740573, 0.024993675, -0.07933814, 0.0064090686, 0.032578446, 0.032940682, -0.012422266, -0.079072, 0.008427305, 0.094235614, 0.080659576, -0.040665917, -0.08650399, 0.031105854, 0.020814111, 0.06437881, 0.010438581, 0.06726853, -0.011356185, 0.00449933, 0.050619584, -0.0041684816, -0.0037916205, -0.00862209, -0.036069583, -0.033092856, -0.08996839, 0.12527077, 0.05365839, -4.848567e-08, 0.05937584, -0.06420152, -0.016957618, 0.0072025787, 0.059579283, -0.002430824, 0.041962486, -0.0506395, -0.058409754, 0.114983946, -0.012802282, -0.02817539, -0.06369224, -0.055873036, 0.021985201, 0.027643684, -0.016343528, 0.020413827, -0.024268107, -0.019330945, 0.11020154, -0.031858914, 0.039179202, 0.021734178, 0.026570005, -0.05292101, 0.075765245, -0.02957554, 0.037306145, 0.031732593, -0.0484194, 0.03519231, -0.06711979, -0.037277885, 0.090817496, 0.022939693, -0.019892158, 0.024260553, 0.024938457, -0.007078374, -0.010788959, 0.010601572, 0.05336603, -0.0007359, -0.07832101, 0.05739783, -0.05445489, -0.038015023, 0.0633694, -0.0003000535, 0.025854977, -0.068209656, 0.088410355, 0.011995063, -0.006621311, 0.040278014, -0.009107122, 0.001905905, 0.016178312, -0.0037381025, -0.035165116, -0.039213505, 0.1156438, 0.106257774], Metadata: {'content': 'Digg \n/circle6 (http://digg.com/linux_unix )—Some say that Digg.com has become more \npopular than Slashdot for providing articles relating to Linux. You can vote on which \narticles are most interesting to you to gain more exposure for an article.\nGroklaw/circle6  (www.groklaw.net )—The place to look for information regarding legal \nissues surrounding open source software.Linux Today\n/circle6  (www.linuxtoday.com )—This site gathers news that is of particular \ninterest to software developers and IT managers.LWN.net\n/circle6  (www.lwn.net )—Produces a weekly newsletter covering a range of Linux \ntopics.\n85057c02.indd   2385057c02.indd   23 10/18/09   11:26:05 PM10/18/09   11:26:05 PM\n24Part I: Getting Off the Ground with Linux\nNewsforge/circle6  (www.newsforge.com )—Bills itself as the “Online Newspaper for Linux \nand Open Source.” Contains many original articles, as well as links to up-to-the-minute \nopen source stories from other locations on the Web.\nLinuxInsider/circle6  (www.linuxinsider.com )—Covers news articles related to Linux \nissues around the world.Linux at Wikipedia \n/circle6 (http://en.wikipedia.org/wiki/Linux )—Contains an \nexcellent write-up of what Linux is, and includes other Wikipedia links to related top-\nics, companies, and issues. Also, provides a good understanding of Linux history and relationships.\nLinux.com\n/circle6  (www.linux.com )—Provides Linux information, news, tips, articles, and \nreference material.CertCities.com\n/circle6  (http://certcities.com/certs/linux_unix/columns/) —\nRegularly publishes articles on Linux and UNIX.\nIf you need help or have questions about Linux, here are a few sites to try:\nLinux Questions/circle6  (www.linuxquestions.org )—In addition to offering forums on \ndifferent Linux distributions, this is a great place to ask questions related to hardware \ncompatibility, security, and networking. The site also has some good tutorials, as well as reviews of books and Linux distributions.\nGoogle Linux\n/circle6  (www.google.com/linux )—Search for Linux-specifi  c information from \nthis part of the Google search site.Linux Forums\n/circle6  (www.linuxforums.org )—Contains active forums on your favorite \ndistributions and has active IRC channels as well.The Linux Documentation Project\n/circle6  (www.tldp.org )—Offers a wide range of \nHOWTOs, guides, FAQs, man pages, and other documentation related to Linux.Linux Help\n/circle6  (www.linuxhelp.net )—Offers forums, news, and current information \nabout the Linux kernel. Also contains information about fi  nding Linux mailing lists, \nnewsgroups, and user groups.Linux Online\n/circle6  (www.linux.org )—Provides a central source of information related to \nLinux distributions, documentation, books, and people.Linux Kernel Archives\n/circle6  (www.kernel.org )—The primary site for Linux kernel devel-\nopment. You can get the latest stable or testing versions of the Linux kernel. Not the fi  rst \nplace to start with Linux, but I thought you would want to know it’s there.\nMajor Linux projects\nAs you may know, the name Linux comes from the Linux kernel created by Linus Torvalds. The \ndesktop, application, server, and other software needed to create a full Linux system are added \n85057c02.indd   2485057c02.indd   24 10/18/09   11:26:05 PM10/18/09   11:26:05 PM\n25Chapter 2: Linux Projects, Activities, and Careers\nfrom other open source projects. The following is a list of some of the major open source software \norganizations that usually have software included with Linux:\nFree Software Foundation/circle6  (www.fsf.org )—Supports the GNU project, which pro-\nduces much of the software outside the kernel that is associated with Linux. In particu-lar, open source versions of nearly every early UNIX command have been implemented by the GNU project. \nApache Software Foundation', 'page': 92, 'source': 'linux_bible.pdf'}

Please refer to attached screenshot regarding the data insertion into Qdrant
qdrant_data_insertion

Special Note

Utilization of batches is still in progress, (the code is already present in the MR) and we are actively working on it to further enhance the efficiency of the integration.

📚 Documentation preview 📚: https://griptape--812.org.readthedocs.build//812/

@collindutter
Copy link
Member

collindutter commented May 31, 2024 •
edited
Loading

Thanks for the PR! Some of the team is traveling right now but we'll try to take a look next week once tests are passing.

@codecov Codecov
Copy link

codecov bot commented May 31, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 87.93103% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ptape/drivers/vector/qdrant_vector_store_driver.py 87.71% 3 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

@hkhajgiwale hkhajgiwale changed the title Seamless Integration of Griptape with Qdrant VectorDB for Enhanced Vector Search Add support for Qdrant VectorDB May 31, 2024
collindutter
collindutter requested changes Jun 3, 2024
View reviewed changes
docs/griptape-framework/drivers/qdrant_vector_store_driver.md Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Show resolved Hide resolved
poetry.lock Outdated Show resolved Hide resolved
tests/unit/drivers/vector/test_qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
@hkhajgiwale hkhajgiwale requested a review from Anush008 June 26, 2024 06:02
Anush008
Anush008 approved these changes Jun 26, 2024
View reviewed changes
Copy link
Contributor

@Anush008 Anush008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @hkhajgiwale.
LGTM with the CI green.

@hkhajgiwale
Copy link
Contributor Author

@collindutter @vasinov Can you please approve this MR?

@hkhajgiwale hkhajgiwale reopened this Jun 26, 2024
@hkhajgiwale
Copy link
Contributor Author

Thank you @hkhajgiwale. LGTM with the CI green.

Thanks @Anush008 for investing time and providing the valuable feedback

@collindutter collindutter requested review from vasinov and vachillo June 26, 2024 14:35
vachillo
vachillo requested changes Jun 26, 2024
View reviewed changes
Copy link
Member

@vachillo vachillo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks great, there's some changes needed for the example in the docs. all code snippets in the docs run as integration tests on a remote runner. the test needs to be updated to support that.

docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
docs/griptape-framework/drivers/vector-store-drivers.md Outdated Show resolved Hide resolved
griptape/drivers/vector/qdrant_vector_store_driver.py Outdated Show resolved Hide resolved
@hkhajgiwale hkhajgiwale requested a review from vachillo July 1, 2024 06:58
@hkhajgiwale
Copy link
Contributor Author

@vachillo @vasinov @collindutter Incorporated all the said changes. Kindly review

@hkhajgiwale
Copy link
Contributor Author

overall looks great, there's some changes needed for the example in the docs. all code snippets in the docs run as integration tests on a remote runner. the test needs to be updated to support that.

Fixed

vachillo
vachillo reviewed Jul 1, 2024
View reviewed changes
Copy link
Member

@vachillo vachillo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! one last request: please add

- `QdrantVectorStoreDriver` to integrate with Qdrant vector databases.

under Added in the CHANGELOG

@hkhajgiwale hkhajgiwale requested a review from vachillo July 2, 2024 08:12
@hkhajgiwale
Copy link
Contributor Author

looks great! one last request: please add

- `QdrantVectorStoreDriver` to integrate with Qdrant vector databases.

under Added in the CHANGELOG

Added in the changelog.

@vachillo vachillo mentioned this pull request Jul 2, 2024
Merged
3 tasks
@vachillo vachillo closed this in #928 Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Anush008 Anush008 Anush008 approved these changes

@vachillo vachillo vachillo approved these changes

@collindutter collindutter Awaiting requested review from collindutter

@vasinov vasinov Awaiting requested review from vasinov

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hkhajgiwale @collindutter @Anush008 @vachillo