nielsr HF Staff commited on
Commit
36d4413
·
verified ·
1 Parent(s): 335c9dd

Add evaluation results from Github README

Browse files

This PR adds the evaluation results on APEval, EvalPlus, CanItEdit and OctoPack from the Github README to the model card, making it easier for users to understand the performance of the model.

Files changed (1) hide show
  1. README.md +54 -9
README.md CHANGED
@@ -1,11 +1,11 @@
1
  ---
2
- tags:
3
- - code
4
  base_model:
5
  - TechxGenus/CursorCore-QW2.5-1.5B
6
  library_name: transformers
7
- pipeline_tag: text-generation
8
  license: apache-2.0
 
 
 
9
  ---
10
 
11
  # CursorCore: Assist Programming through Aligning Anything
@@ -48,6 +48,16 @@ CursorCore is a series of open-source models designed for AI-assisted programmin
48
 
49
  Our models have been open-sourced on Hugging Face. You can access our models here: [CursorCore-Series](https://huggingface.co/collections/TechxGenus/cursorcore-series-6706618c38598468866b60e2"). We also provide pre-quantized weights for GPTQ and AWQ here: [CursorCore-Quantization](https://huggingface.co/collections/TechxGenus/cursorcore-quantization-67066431f29f252494ee8cf3)
50
 
 
 
 
 
 
 
 
 
 
 
51
  ## Usage
52
 
53
  Here are some examples of how to use our model:
@@ -114,13 +124,27 @@ sample = {
114
  {
115
  "type": "code",
116
  "lang": "python",
117
- "code": """def quick_sort(arr):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quick_sort(left) + middle + quick_sort(right)"""
 
 
 
 
 
 
 
118
  }
119
  ],
120
  "current": {
121
  "type": "code",
122
  "lang": "python",
123
- "code": """def quick_sort(array):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quick_sort(left) + middle + quick_sort(right)"""
 
 
 
 
 
 
 
124
  },
125
  "user": ""
126
  }
@@ -202,7 +226,14 @@ sample = {
202
  "current": {
203
  "type": "code",
204
  "lang": "python",
205
- "code": """def quick_sort(array):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quick_sort(left) + middle + quick_sort(right)"""
 
 
 
 
 
 
 
206
  },
207
  "user": "Add Docstring."
208
  }
@@ -273,7 +304,14 @@ sample = {
273
  "current": {
274
  "type": "code",
275
  "lang": "python",
276
- "code": """def quick_sort(array):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quick_sort(left) + middle + quick_sort(right)"""
 
 
 
 
 
 
 
277
  },
278
  "user": "Add Docstring."
279
  }
@@ -342,7 +380,14 @@ sample = {
342
  "current": {
343
  "type": "code",
344
  "lang": "python",
345
- "code": """def quick_sort(array):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) // 2]\n left = [x for x in arr if x < pivot]\n middle = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quick_sort(left) + middle + quick_sort(right)"""
 
 
 
 
 
 
 
346
  },
347
  "user": "Add Docstring."
348
  }
@@ -414,4 +459,4 @@ CursorCore is still in a very early stage, and lots of work is needed to achieve
414
 
415
  ## Contribution
416
 
417
- Contributions are welcome! If you find any bugs or have suggestions for improvements, please open an issue or submit a pull request.
 
1
  ---
 
 
2
  base_model:
3
  - TechxGenus/CursorCore-QW2.5-1.5B
4
  library_name: transformers
 
5
  license: apache-2.0
6
+ pipeline_tag: text-generation
7
+ tags:
8
+ - code
9
  ---
10
 
11
  # CursorCore: Assist Programming through Aligning Anything
 
48
 
49
  Our models have been open-sourced on Hugging Face. You can access our models here: [CursorCore-Series](https://huggingface.co/collections/TechxGenus/cursorcore-series-6706618c38598468866b60e2"). We also provide pre-quantized weights for GPTQ and AWQ here: [CursorCore-Quantization](https://huggingface.co/collections/TechxGenus/cursorcore-quantization-67066431f29f252494ee8cf3)
50
 
51
+ We use the manually written benchmark APEval to assess the model's ability to assist programming. We also utilize [EvalPlus](https://github.com/evalplus/evalplus), [CanItEdit](https://github.com/nuprl/CanItEdit) and [OctoPack](https://github.com/bigcode-project/octopack) to evaluate the model's performance in Python program generation, instructional code editing, and automated program repair. Since we use a custom conversation template, its generation method differs significantly from both instruct models and base models. Please refer to [our paper](http://arxiv.org/abs/2410.07002) for more details.
52
+
53
+ Evaluation results on APEval:
54
+
55
+ <img src="https://github.com/TechxGenus/CursorCore/blob/main/pictures/APEval.png" alt="APEval" width="75%"/>
56
+
57
+ Evaluation results on EvalPlus, CanItEdit and OctoPack:
58
+
59
+ <img src="https://github.com/TechxGenus/CursorCore/blob/main/pictures/EvalPlus_CanItEdit_OctoPack.png" alt="EvalPlus_CanItEdit_OctoPack" width="75%">
60
+
61
  ## Usage
62
 
63
  Here are some examples of how to use our model:
 
124
  {
125
  "type": "code",
126
  "lang": "python",
127
+ "code": """def quick_sort(arr):
128
+ if len(arr) <= 1:
129
+ return arr
130
+ pivot = arr[len(arr) // 2]
131
+ left = [x for x in arr if x < pivot]
132
+ middle = [x for x in arr if x == pivot]
133
+ right = [x for x in arr if x > pivot]
134
+ return quick_sort(left) + middle + quick_sort(right)"""
135
  }
136
  ],
137
  "current": {
138
  "type": "code",
139
  "lang": "python",
140
+ "code": """def quick_sort(array):
141
+ if len(arr) <= 1:
142
+ return arr
143
+ pivot = arr[len(arr) // 2]
144
+ left = [x for x in arr if x < pivot]
145
+ middle = [x for x in arr if x == pivot]
146
+ right = [x for x in arr if x > pivot]
147
+ return quick_sort(left) + middle + quick_sort(right)"""
148
  },
149
  "user": ""
150
  }
 
226
  "current": {
227
  "type": "code",
228
  "lang": "python",
229
+ "code": """def quick_sort(array):
230
+ if len(arr) <= 1:
231
+ return arr
232
+ pivot = arr[len(arr) // 2]
233
+ left = [x for x in arr if x < pivot]
234
+ middle = [x for x in arr if x == pivot]
235
+ right = [x for x in arr if x > pivot]
236
+ return quick_sort(left) + middle + quick_sort(right)"""
237
  },
238
  "user": "Add Docstring."
239
  }
 
304
  "current": {
305
  "type": "code",
306
  "lang": "python",
307
+ "code": """def quick_sort(array):
308
+ if len(arr) <= 1:
309
+ return arr
310
+ pivot = arr[len(arr) // 2]
311
+ left = [x for x in arr if x < pivot]
312
+ middle = [x for x in arr if x == pivot]
313
+ right = [x for x in arr if x > pivot]
314
+ return quick_sort(left) + middle + quick_sort(right)"""
315
  },
316
  "user": "Add Docstring."
317
  }
 
380
  "current": {
381
  "type": "code",
382
  "lang": "python",
383
+ "code": """def quick_sort(array):
384
+ if len(arr) <= 1:
385
+ return arr
386
+ pivot = arr[len(arr) // 2]
387
+ left = [x for x in arr if x < pivot]
388
+ middle = [x for x in arr if x == pivot]
389
+ right = [x for x in arr if x > pivot]
390
+ return quick_sort(left) + middle + quick_sort(right)"""
391
  },
392
  "user": "Add Docstring."
393
  }
 
459
 
460
  ## Contribution
461
 
462
+ Contributions are welcome! If you find any bugs or have suggestions for improvements, please open an issue or submit a pull request.