Dataset to replace daily dialog

.github/workflows/scorecard.yaml

@@ -10,7 +10,7 @@ on:
   # To guarantee Maintained check is occasionally updated. See
   # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained
   schedule:
-    - cron: '44 15 * * 0'
+    - cron: "44 15 * * 0"
   # push:
   #   branches: [ "main", "covassist-cleanup" ]
   # pull_request:

.gitignore

@@ -23,3 +23,5 @@ interfaces/Demos/prompter/
 coverage.xml
 convassist/.coverage.*
 convassist/tests/test_data/*
+3rd_party_resources/human-conversation/Human_Conversation_Data.txt
+3rd_party_resources/personachat/Persona_Chat_Data.txt

.vscode/launch.json

@@ -1,23 +1,18 @@
 {
     "version": "0.2.0",
     "configurations": [
-        // {
-        //     "name": "Python Debugger: Module",
-        //     "type": "debugpy",
-        //     "request": "launch",
-        //     "module": "pyinstaller"
-        // },
         {
-            "name": "Python Debugger: Current File",
+            "name": "Python Debugger: Main with Arguments",
             "type": "debugpy",
             "request": "launch",
             "program": "${file}",
-            "console": "internalConsole",
-            "justMyCode": false,
-            "redirectOutput": true,
+            "console": "integratedTerminal",
+            "cwd": "${workspaceFolder}",
             "env": {
-                "PYTHONPATH": "${workspaceFolder}"
-            }
+                "PYTHONPATH": "${workspaceFolder}/src/"
+            },
+            "justMyCode": false,
+            "args": "${command:pickArgs}"
         },
         {
             "name": "Continuous Predict Util",

3rd_party_resources/README.md

@@ -0,0 +1,34 @@
+# ConvAssist 3rd Party Resources
+
+## Datasets
+
+TODO: Explanation of why we need conversation datasets for the predictors
+
+Two datasets are provided by ConvAssist:
+* [Human Conversation Training Data](human-conversation/README.md)
+* [Personachat](personalchat/README/md)
+
+### Licensing
+
+> [!NOTE]
+> Datasets included in the ConvAssist project are subject to the license provided.  We make no
+> claim of copyright to the data provided by.  See specific license information for each
+> dataset.
+
+### "Bring your own conversation dataset"
+
+If you would like to extend ConvAssist with your own converstaion dataset:
+* Conversation data **MUST** be in plain text format.
+* Each sentence should be on a separate line.
+* ConvAssist expects the conversation data to be in a single file.  You may combine multiple datasets to provide a more robust library to the predictors.
+
+The following Predictors support custom conversation dataset(s)
+
+| Predictor | .ini Setting |
+| --- | --- |
+| GeneralWordPredictor| aac_dataset |
+| SmoothedNGramPredictor* | N/A |
+| SentencePredictor | retrieve_database |
+
+>\* NOTE
+The SmoothedNGramPredictor uses an AAC like dataset to populate the NGram database used for predictions.  ConvAssist uses the data from [Human Conversation Training Data](human-conversation/README.md) in the [ACATConvAssist](../interfaces/ACAT/acatconvassist/) utility as well as the [Continuous Predictor](../interfaces/Demos/continuous_predict/) Demo application.

3rd_party_resources/human-conversation/LICENSE

@@ -0,0 +1,119 @@
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

3rd_party_resources/human-conversation/README.md

@@ -0,0 +1,28 @@
+# Human Conversation Training Data
+Training data aggregated from various sources for training a chatbot with NLP.
+
+## License
+[CC0 1.0 Universal](LICENSE)
+
+## About Dataset
+Original Dataset Card from [projjail1/Human-Conversation-Training-Data](https://www.kaggle.com/datasets/projjal1/human-conversation-training-data/data)
+
+> **Dataset Card**
+> ### Context
+> 
+> I was working with RNN models in Tensorflow and was searching about conversation bots. Then a idea struck me as to create a bot myself. I looked for chat data but was not able to find something useful. Then I came across Meena chatbot and Mitsoku chatbot data and so compiled them with some data from human chats corpus.
+> 
+> ### Content
+> 
+> The data corpus contain chat labelled chat data with Human 1 and Human 2 in ask-reponse manner.
+> Each odd row with Human 1 label is the initiator of the chat and each even row with Human 2 label is the response.
+> Data after Human x: is the chat data which can be preprocessed to remove the label part.
+> 
+> ### Acknowledgements
+> 
+> We wouldn't be here without the help of others. If you owe any attributions or thanks, include them here along with any citations of past research.
+> 
+> ### Inspiration
+> 
+> I would love others to explore this data and frame ideas related to the creation of a chatbot system.
+